diff options
Diffstat (limited to 'posts/umass-ctf-2021-postmortem.org')
| -rw-r--r-- | posts/umass-ctf-2021-postmortem.org | 342 |
1 files changed, 342 insertions, 0 deletions
diff --git a/posts/umass-ctf-2021-postmortem.org b/posts/umass-ctf-2021-postmortem.org new file mode 100644 index 0000000..2bc284c --- /dev/null +++ b/posts/umass-ctf-2021-postmortem.org @@ -0,0 +1,342 @@ +#+TITLE: UMass CTF 2021 Postmortem +#+DATE: <2021-04-19 Mon 10:24> +#+TAGS: writeup capture-the-flag + +This was the first year our capture-the-flag event, [[https://ctftime.org/event/1282][UMass CTF 2021]], was open to +the public. The competition started Friday, March 26th at 18:00 EDT, and ended +Sunday, March 8th at the same time. By the end of the competition, we had *1991 +registered users*, belonging to *1160 registered teams*. No teams were tied, we had +just one unsolved challenge, and each of the "harder" challenges had just one or +two solves. + +In this post, I will be reflecting on what we did well, and how we intend to +improve for next year's UMass CTF. + +But, before I begin, I realize that some may not know *what a capture-the-flag +(CTF) is*. If you need an explanation, I will direct you to [[https://dev.to/atan/what-is-ctf-and-how-to-get-started-3f04][this article]]. + +A huge thank you to the rest of the [[https://umasscybersec.org/][UMass Cybersecurity Club]], especially those +on infrastructure who worked around the clock to help me fix challenges. Thank +you to my beautiful, loving partner, who helped me come up with the names and +flags for quite a few challenges. Thank you to my loving parents, who hosted me +while I worked on this, my degree, and my research. Thank you to my best friend, +who made some challenges when I mentioned the competition to him. + +Table of Contents: + +- [[*Challenge Design][Challenge Design]] + - [[*replme (and replme2)][replme + replme2]] + - [[*babushka][babushka]] + - [[*chains][chains]] + - [[*scarymaze][scarymaze]] + - [[*stegtoy][stegtoy]] + - [[*warandpieces][warandpieces]] + - [[*suckless2][suckless2]] + - [[*easteregg][easteregg]] + - [[*lambda][lambda]] +- [[*Lessons Learned][Lessons Learned]] + - [[*Playtesting a Priority][Playtesting a Priority]] + - [[*Obvious Solutions Should Be Obvious][Obvious Solutions Should Be Obvious]] + - [[*Challenge Update Transparency][Challenge Update Transparency]] + - [[*Matchmaker][Matchmaker]] + +* Challenge Design + +I'll get one thing out of the way: writing challenges *will take much more time +than you're expecting it to*. I was asked to come up with ten or so challenges +about three months before we were supposed to go live. I thought I'd be able to +get them done over the course of a weekend before spring semester started. In +reality, I was working on challenges up to the day of. + +Below are some thoughts on the challenges I made, from what I would consider +"most interesting" to "least interesting". Nearly every challenge had something +go wrong with it. + +# At the time I began working on challenges, I hadn't played in a CTF in almost a +# year. I was working off a combination of old memories and vague ideas of +# situations I would expect to see in the real world. It's possible challenge +# creation would not have taken as much time had I been more active in the +# community at the time. + +# a handful of people mentioned that they enjoyed playing in our competition +# much more than in the [[https://ctftime.org/ctf/52][VolgaCTF qualifier]], which was running at the same time. + +** replme (and replme2) + +This was my favorite challenge. + +Writeups: +- [[https://anee.me/janet-v1-1-repl-sandbox-bypass-f8bbe141a4f1][Janet v1.1 REPL Sandbox Bypass]] (by [[https://twitter.com/lionaneesh][Aneesh Dogra]]) +- [[https://thegoonies.github.io/2021/03/28/umass-ctf-2021-replme/][UMass CTF 2021 - replme [pwn]]] (by [[https://thegoonies.github.io/2021/03/28/umass-ctf-2021-replme/][TheGoonies]]) +- [[https://yichenchai.github.io/blog/umassctf-replme][UMassCTF'21 replme writeup]] (by [[https://yichenchai.github.io/blog/umassctf-replme][Chai Yi Chen]]) + +Older versions of the [[https://janet-lang.org/][Janet programming language]] had a vulnerable [[https://leonardschuetz.ch/blog/nan-boxing/][NaN boxing]] +implementation. An attacker capable of running arbitrary Janet code could write +specific bytes to a =float64= object, resulting in type confusion. + +The challenge presented itself as an [[https://replit.com/][online REPL]] for Janet[fn:1] that +blacklisted functions dealing with the filesystem. The blacklisting was done by +overwriting the functions with one that would error (i.e. =(defn os/shell [&] +(error ...=). The /intended/ solution was to use the NaN boxing vulnerability to +create a =cfunction= object that pointed at the original implementation. + +The reason there was a replme2 is that I didn't blacklist all of the relevant +functions. At least one team realized that they could call =(slurp "flag.txt")=. +This is something that [[*Playtesting a Priority][playtesting by the rest of the team]] likely would have +caught, but I'll save that for a later section. Sunday morning, I released a +fixed version. Some teams solved it instantly, indicating that they had either +found a different unintended solution, or that they had solved replme the +intended way. + +** babushka + +A Python bytecode crackme. The challenge was titled "babushka" because the +program is designed like a [[https://en.wikipedia.org/wiki/Matryoshka_doll][matryoshka doll]]. There is one entry point to a long +chain of functions, each one unpacking and calling the next in the chain. + +Each of these functions does some check on the input. All of them, except for +the fourteenth, have some check that involves a decoy flag, and some way of +combining the results of the checks further down. They are combined such that +only the output of one of the 500 functions is used. + +The intended solution was to write a script to extract all of the functions, and +look at the code for combining function outputs to see which mattered. You could +then manually reverse engineer that function to get the flag. + +The way the input is checked is about the same across all functions, so in +theory, one could write a script to extract the value the input is checked +against from all of the functions, and then try all 500 to see which one worked. + +I [[https://git.sr.ht/~jakob/umass-ctf-2021-challenge-writeups/tree/master/item/babushka/gen.py][programmatically generated]] and obfuscated the script that was given to teams. +This challenge was fairly unremarkable, besides there being no writeups, and +some people giving me flak because it didn't work on their version of Python. + +Oh well. I had a lot of fun making it. + +** chains + +Writeups: +- [[https://ctftime.org/writeup/26868][chains]] (by [[https://ctftime.org/user/48335][Chris Greene]]) +- [[https://scavengersecurity.com/posts/umass-chains/][UMassCTF 2021 - Chains [Reversing]]] (by [[https://scavengersecurity.com/][Scavenger Security]]) +- [[https://github.com/1GN1tE/CTF_Writeups/tree/main/Writeups/UMassCTF_2021/Chains][Chains]] (by [[https://github.com/1GN1tE][Arijeet Mondal]]) + +This was an "optimizeme" challenge, one where the flag is generated by an +inefficient algorithm. To get the flag in a reasonable amount of time, one would +need to either patch the algorithm or reimplement it. + +Here, the characters of the flag were encoded as very large numbers, and the +program would, for each character, generate the entire [[https://en.wikipedia.org/wiki/Collatz_conjecture][Collatz sequence]] to see +where that number occurs. There were some pretty clever solutions, but the one I +used when I was playtesting was to memoize the Collatz function. + +Oh, and to make it less reasonable for someone to let the program run for the +duration of the competition, I compiled for AARCH64. I apologize to the handful +of people who were unable to get an ARM environment set up.[fn:3] + +** scarymaze + +This was my second favorite challenge, even though a lot went wrong with it. + +This was a networked maze game written in [[https://www.rust-lang.org/][Rust]], where the server would +spit out the flag after 500 mazes had been solved. I spent a lot of time on the +packet format and the renderer. + +#+CAPTION: Raycaster with "hacker terminal green" walls. +[[./umass-ctf-2021-postmortem-1.png]] + +Communications between the client and the server were encrypted. The algorithm +was AES-128-CBC with the key =STRINGS NOT HERE=.[fn:4] I made things slightly +easier by using OpenSSL for the AES implementation, and slightly harder by +linking statically. If I'd linked dynamically, it would have been trivial to run +the client with [[https://www.ltrace.org/][ltrace]]. Static linking should've helped somewhat, since the +OpenSSL =.a= includes a ton of informative strings, but no one came close to a +solution until I'd re-released the client binary with symbols. + +And then, in the last 30 minutes or so of the competition, two people had +scripts, but the server was crashing around 200 mazes in. I couldn't figure out +the bug in that short period of time, so I hastily re-deployed with a server +that would spit out the flag after 50 solves. It was stressful. + +Small lesson: networked games, where someone has to reimplement the protocol, +should be somewhat tolerant to errors. In this case, the server panicked and +closed the connection if there was /anything/ wrong with the header, which I'll +admit is bullshit. + +** stegtoy + +The idea of leaking memory through a visual medium is neat to me, so I made a +[[https://git.sr.ht/~jakob/umass-ctf-2021-challenge-writeups/tree/master/item/stegtoy/bmpencode.c][shitty]] BMP utility that encodes text into the least-significant bits of the +image data. Source code wasn't made available during the competition, as I find +leaking memory blindly to be more interesting. + +This seemed to make it a very difficult challenge. Only one team solved it, +which surprised me. BMP is [[https://github.com/corkami/pics/blob/master/binary/BMP.png][simple]] enough that there are only so many things one +could have tried. Anyway, the intended solution was to overwrite the "data +start" pointer in some BMP file so that the tool would leak the heap data +containing the flag. + +** warandpieces + +Writeups: +- [[https://ctftime.org/writeup/26805][War and Pieces]] (by [[https://ctftime.org/user/106112][legallybearded]]) +- [[https://ctftime.org/writeup/26803][War and Pieces]] (by [[https://ctftime.org/user/68391][ugotjelly]]) + +This wasn't one I had originally planned. I was in the grocery store one day and +saw a bag of toy army soldiers for $1.99. I bought them because that was one of +my favorite things to play with when I was younger. My mother, upon seeing them, +asked if I had gotten them for the CTF, which gave me the idea. + +The pieces came in 6 poses and 2 colors. I encoded the flag as hexadecimal, and +assigned each combination of pose/color/orientation to a hexadecimal nibble. In +retrospect, the challenge would've been more interesting had I treated the flag +as a natural number, and written it in base-12 using the piece alphabet. + +The flag was supposed to be =UMASS{lil_t0y_s0lj4s}=, but I made a mistake when +lining up the pieces: in the script I had made to help me line up the toys for +the picture, I had accidentally used the same color for two different digits. +Someone was kind enough to point it out, so we accepted the mangled flag +(=UMASS{lfl_t0v_s0lj4s}=). This wasn't a great decision, though, since you +wouldn't realistically be able to pull that out of list of guesses. I should +have just done the picture over again. + +I'm hoping that next time, I won't be working on challenges in the week before +the competition. Here, I had only playtested the first few steps because I was +so short on time, hence this mistake falling through the cracks. + +** suckless2 + +Writeups: +- [[https://drowsycat256.tistory.com/3][suckless2 writeup]] (by [[https://drowsycat256.tistory.com/][renia256]]) + +I have little to say. Someone asked if I could recycle some of my challenges +from last year, but I had already posted a [[http://jakob.space/blog/umass-ctf-2020-writeup.html][writeup]]. I ended up modifying the +challenge in a way that would render my exploit useless, and releasing that as a +challenge. It was meant to be easy. + +** easteregg + +Writeups: +- [[https://szymanski.ninja/en/ctfwriteups/2021/umassctf/easteregg/][UMassCTF'21: easteregg]] (by [[https://szymanski.ninja/en/about/][Łukasz Szymański]]) +- [[https://gitlab.com/hacklabor/ctf/writeup/-/blob/master/reverse/easteregg_umass21/easteregg_.md][easteregg]] (by [[https://hacklabor.de/][Th0m4sK]]) +- [[https://www.youtube.com/watch?v=l9zRBJJYTgg][Easteregg UMASS CTF 21]] (by [[https://twitter.com/hacklabor][Hacklabor]]) +- [[https://www.youtube.com/watch?v=Ao5r6ptvKGc][Engenharia reversa - UMass CTF'21 - 'Easteregg' Writeup]] (by [[https://twitter.com/m0v_ebp_esp][zapzap kkkkkkkkkkkkkkkkkkkkk '-']]) + +Like warandpieces, this was not a challenge I had originally planned to make. +One of the problem sets for the introduction to computer systems class this +semester involved reverse engineering an ELF binary. In an effort to appeal to +students who actually attend UMass, I was asked to make an easy reverse +engineering challenge in the same vain. + +I took that class two years ago, and I still had the programs I'd written for +it, so I took one of them (a text adventure game), added an "easter egg" that +you would need to reverse engineer the executable to find, and uploaded it. + +This was one of the most solved challenges. + +** lambda + +Next year, no more compiled Haskell. + +I spent a lot of time on a cool sine scroller with one of my [[https://www.youtube.com/watch?v=gMTyMetv5Sk][favorite mod files]] +playing in the background. + +#+CAPTION: The dreaded 'lambda' challenge. +[[./umass-ctf-2021-postmortem-2.png]] + +And then I made the mistake of writing the key check algorithm in Haskell. This +is a reminder that there's a fine line between "possible" and "reasonable". No +one solved this. + +When I'm done with school in two weeks, I'll give this my all, and give it a +proper write-up. + +* Lessons Learned + +Now, I'll share some specific morsels of wisdom that came to me after this very +stressful weekend. + +** Playtesting a Priority + +This might be seem obvious, but please humor me. + +I made an effort to go through all of my challenges to see that they were +reasonable, and I suspect this was the case for the rest of the team as well. It +was better than nothing, but we could have done better in a couple of ways. I'll +use three challenges as examples: [[*replme (and replme2)]['replme']] and [[*warandpieces]['warandpieces']], challenges I had +created, and 'pikcha', made by Steven. + +The original [[*replme (and replme2)]["replme"]] was solvable by one function that wasn't blacklisted. I +think a less-experienced player would have come across it, and we do have a +handful of people on our team who are less familiar with the "pwn" category. + +When playtesting [[*warandpieces]["warandpieces"]], I only went through the first line or so. The +error doesn't show up until much later. If I had either solved it in full, or +had someone else playtest it, I would have caught the error. + +"pikcha" included the answer in the session. LOL. I think if I had playtested it +before we went live, I would have caught it. + +These issues could have been caught if /we playtested our peer's challenges/. + +This is what I have proposed to do next time: for every challenge, we choose two +other people to playtest it. That way, there are three (if you include the +challenge creator) pairs of eyes on the challenge. The only work on challenges +happening in the last month should be fixing issues that come up during +playtesting. Playtesters should also be tracking how much time they spend on +each challenge. + +** Obvious Solutions Should Be Obvious + +Often, a CTF will have an easy "entry" challenge that doesn't require any +solving. The flag might be given in the challenge description. In our case, it +was hidden /somewhere/ in the communication platform we used. As an idea, it looks +good on paper, but actually implementing the challenge was one of the biggest +mistakes we made. We had nearly a hundred people spamming some variation of +"/flag" or "!flag" into the chat, and a handful of people opening support +tickets because they thought the flag was supposed to be obtained by talking to +us. + +A few minutes in, someone made the following joke: + +#+BEGIN_SRC prog +<Youssef>: UMASS{WowIReallySuck...} +<ralpmeTS>: !flag +<Youssef>: I actually hate myself... +<Youssef>: I found the flag. +#+END_SRC + +Besides 'UMASS{flag}', which is what we had in the message describing the flag +format, 'UMASS{WowIReallySuck...}' was the second most popular incorrect answer +for this challenge. + +It was a mess. No more challenges like this.[fn:2] + +** Challenge Update Transparency + +When challenges went down, or were modified, announcements went through our +communication platform. Not everyone saw those announcements, and some teams had +an unfair advantage by getting both broken and fixed versions of the same +challenge. + +Next year, announcements will occur on the CTF platform as well. And hopefully, +with our commitment to playtesting, we will not need to fix any challenges after +going live, but we will ensure the old versions are still available if we do. + +** Matchmaker + +Some people seemed to still be looking for teams in the week leading up to the +competition, so I put together a little form where people would enter their +experience, timezone, etc. + +It was a good idea, but I think only about one team came about from this, since +a lot of people either put bad contact information or flaked. + +Next year, the matchmaking form will be available at the time we announce the +competition, and the matchmaking process will be partially automated. + +--- + +[fn:1] It isn't really a REPL. Just a =textarea= for a Janet script, and a button that runs it, showing the output in a different =textarea=. +[fn:2] I honestly didn't like the idea in the first place, because I hate Discord and don't want to feel as though I'm forcing people to use it if they don't already. But I digress... +[fn:3] Someone mentioned being on mobile data, and that QEMU would have been prohibitively large for his plan. I felt terrible. +[fn:4] Surprisingly, nobody asked me about this. I thought this would have been a dead giveaway. |